Build log

Running a 284-Billion-Parameter Model on a 121GB Box

Tyler Merritt · August 2, 2026

I spent the week fighting a 284-billion-parameter model into a 121GB machine, and the whole exercise came down to one question: does a model this size run usefully on a DGX Spark, or is it a party trick? The answer is "yes, barely — and here's exactly how."

The hardware and the stack

The box is an NVIDIA GB10 (DGX Spark): 121GB unified memory, compute capability 12.1, running driver 595.71.05 with CUDA 13.2. The model is DeepSeek-V4-Flash-0731, pulled from unsloth/DeepSeek-V4-Flash-0731-GGUF as an IQ3_XXS quantized GGUF — 284.33B total parameters at 3.0625 bits per weight, landing at 97.05 GiB across four shards. That's the whole trick: you can't fit 284B at FP8 (that's ~284GB), so the aggressive 3-bit quantization is the only reason this runs at all.

The runtime is llama.cpp b10235 (commit 221f0f635), CUDA backend, rebuilt on the day of deployment. It's a llama.cpp deployment, not vLLM — and that distinction matters more than you'd think, for reasons I'll get to.

Staging: the part nobody talks about

Before the model even loads, two things have to happen:

  1. Stop both vLLM servers. They were holding 85GB of the 121GB box (Gemma4 MTP at 40.3GB, Qwen3.6 hybrid at 44.7GB).
  2. Drop the page cache — and this was NOT optional. After stopping both servers, the box still reported 89GiB "used" with zero GPU compute apps running. That's the known GB10 stuck-unified-memory condition. Available was only 31GiB — the 97GiB model wouldn't have fit. After drop_caches: 5.3GiB used, 116GiB available. Swap drained from 7.8GiB to 839MiB.

Peak during the run was 102GiB used, 18GiB still free. No OOM. It fits, but it's a tight fit — the kind where you stop holding your breath when the load completes.

Benchmark results (-ngl 999, all layers on GPU)

  • Prefill: 342.80 ± 0.79 tok/s at pp512 (empty context), 314.19 tok/s at pp2048 (16K context) — a −8.3% drop.
  • Decode: 16.74 ± 0.08 tok/s at tg128 (empty context), 15.67 tok/s at 16K context — a −6.4% drop.
  • Context scaling: decode loses under 7% going from empty to 16K of context, so long conversations aren't a cliff.

But the headline number is 16.74 tok/s decode, and that's the whole story of this deployment.

The MTP problem: why there's no draft support

That 16.7 tok/s is the ceiling, and here's the frustrating part — it didn't have to be. Speculative decoding is the single biggest lever on that number, and DeepSeek's architecture is built around MTP (multi-token prediction). But every path to MTP is closed on this setup:

  1. The MTP module isn't in the GGUF. I scanned all four shards: 1,328 tensors, highest block index 42 against block_count = 43, and zero nextn/eh_proj/enorm/hnorm/shared_head tensors. DeepSeek's MTP layer was dropped during GGUF conversion — known llama.cpp convert behavior.
  2. llama.cpp has no DeepSeek-MTP implementation regardless. Its speculative decoding is separate-draft-model only (--spec-draft-hf, -md). There's no deepseek_mtp equivalent.
  3. A conventional draft model is impractical. A draft must share the target tokenizer, and there's no small DeepSeek-V4-vocab GGUF. The cached DeepSeek-R1-Distill-Qwen-7B is Qwen-vocab, so it can't draft for this model.
  4. The path where MTP does exist is closed here. vLLM and SGLang support deepseek_mtp on the original safetensors — but 284B at FP8 is ~284GB and won't fit in 121GB. IQ3_XXS at 97GiB is what makes this run at all, and that quant format is llama.cpp-only.

So 16.7 tok/s is the ceiling. Part of the gap to Gemma4's 65–79 tok/s is that Gemma4 runs MTP at ~3.2 accepted tokens/step; this model has no drafter at all.

The Hermes Agent checks — 64K is a hard floor

This is the part that surprised me. Hermes Agent refuses any model under 64,000 tokens outright. That's not a preference, it's a hard floor in the agent. At the default server parallelism, the window split to n_ctx_slot = 32768, which Hermes rejected — setting --parallel 1 restored n_ctx_slot = 65536 and everything worked.

That single flag was the difference between "deployed" and "rejected." Worth knowing if you ever point an agent at a model with a small context window.

Other things I measured while it ran

  • Requests serialize. 1 request = 9.08s; 2 concurrent = 18.08s wall (B finished at 9.09s, A at 18.07s). Swarm-style workloads queue rather than run in parallel.
  • Reasoning doesn't leak. --jinja cleanly separates reasoning into reasoning_content; the content field is clean, no marker tokens. Not the Gemma4 enable_thinking failure mode. But reasoning is verbose — "What is 2+2? Explain briefly." cost 227 completion tokens, 713 chars of reasoning, ~13.6 seconds for a trivial question.
  • The max_tokens question (still open). Currently 8192 against a 64K window — a 1:8 ratio, where the Gemma4 tuning settled on 1:16. At 16.7 tok/s, a full-length 8192-token response takes ~8 minutes, against Hermes timeouts tuned for a 65–79 tok/s model. 4096 would halve that. Left at 8192 pending my call.

The honest bottom line

This is an exclusive-use model. Running it requires taking BOTH production vLLM servers down — there's no co-residency at 97GiB on a 121GiB box. It can't sit alongside the current stack.

16.74 tok/s decode is roughly a quarter of Gemma4's throughput, for a model with ~11× the total parameters at aggressive 3-bit quantization. Whether that trade is worth it is a quality question this benchmark doesn't answer — and that's the honest thing to say: this answers "does a 284B model run usefully on this box," not "is it faster than Gemma4."

I'm running it as my daily driver for a week to find out. The benchmark says it runs. The next week decides if it's worth it.