# Benchmarking Speculative Decoding on DGX Spark: 11.5 → 29.5 tok/s > Source: > Published: 2026-08-20 18:59:52+00:00 ## Bandwidth, not compute The DGX Spark pairs a GB10 superchip with 128 GB of unified LPDDR5X. Enough room for a 70B model on a desk, but only 273 GB/s to feed it. Running Qwen3.8-27B at Q4_K_XL (17.5 GB of weights), I measured **11.5 tokens per second**. Across eight tasks (Chinese and English prose, code generation and editing, JSON, reasoning, translation, a 2,177-token summary) every median landed between 11.47 and 11.66 tok/s. A 1.7% total spread. The workload doesn’t matter; that’s what a machine looks like when it spends its time waiting on memory instead of computing. Every generated token reads all 17.5 GB of weights. At the published 273 GB/s that caps generation near 15.6 tok/s; measured 11.5 is 74% of the ceiling. No configuration flag fixes arithmetic. Speculative decoding fits this imbalance precisely: verifying *k* drafted tokens costs one forward pass (a single 17.5 GB read), so accepted guesses are paid for with idle compute instead of scarce bandwidth. The thing people complain about, the wasted FLOPS, is exactly what makes the fix work. The same reasoning applies to Strix Halo, Apple Silicon, and any unified-memory design. ## Where 208 tok/s came from llama.cpp’s `--spec-type ngram-mod` drafts from repeated n-grams, no draft model needed. Benchmarked the usual way (same prompt, five runs, median) it reported up to 208 tok/s. But the first request in a fresh process is always ~11.5, and every later one climbs: the n-gram cache **persists across requests inside the server process**, so by run two it contains the complete answer from run one. The benchmark was measuring the cache replaying its own output. Setting `"cache_prompt": false` disables the KV prompt cache. It does **not** touch the n-gram draft cache, and no request-level flag does. **If you benchmark llama.cpp's n-gram speculation by repeating a prompt, your numbers are wrong.** Restart the process or vary the prompt. The honest cold-start figure here is 11.5–12.4 tok/s, no measurable benefit at all. ## Five strategies on one binary Same model, same quantization, same server binary, idle machine; the only variable is the `--spec-type` flag and its draft model. Five runs per cell, median reported, spread ±0.1–2.8%. | Task | Baseline | Draft 2B | DSpark | DFlash2 | DFlash2 × | |---|---|---|---|---|---| | zh-prose | 11.66 | 14.42 | 18.43 | 22.15 | | | en-prose | 11.54 | 13.24 | 17.19 | 21.09 | | | code-gen | 11.54 | 15.54 | 23.89 | 28.06 | | | code-edit | 11.54 | 12.96 | 18.71 | 23.09 | | | json-out | 11.54 | 19.58 | 20.83 | 23.33 | | | reasoning | 11.54 | 23.35 | 26.61 | 28.33 | | | translate | 11.53 | 24.11 | 24.68 | 29.48 | | | long-ctx | 11.47 | 14.05 | 19.63 | 22.02 | | | ngram-mod (cold, honest) is omitted: 11.5–12.4 tok/s, indistinguishable from baseline. | **DFlash2 wins every task**, at 1.83× to 2.56× the baseline. It is a block-diffusion drafter: it emits a whole block of guesses in one pass and traces a coherent path through the candidates, and it’s a 2 GB file sitting next to a 17.5 GB model. Note that its numbers finally *vary* by task (21–29 tok/s) while the baseline was flat: the bandwidth ceiling is no longer the binding constraint. ## Three things the numbers taught me ### Acceptance rate is not comparable across drafter architectures On code generation the sequential 2B drafter accepts **88%** of its guesses and delivers 15.5 tok/s; DFlash2 accepts **75%** and delivers 28.1. Ranking by acceptance rate, the metric every paper reports, picks the slower system by 1.8×. A rejected sequential guess is a whole wasted forward pass; a rejected block position costs almost nothing. When comparing sequential against block drafters, measure wall-clock throughput or nothing. ### Lossless in distribution, not reproducible in practice At temperature 0, DSpark and DFlash2 matched a no-speculation control byte-for-byte on only 6 of 8 tasks (the control itself was fully deterministic). The divergence I traced landed exactly on the sequence’s second-narrowest top-2 logprob margin, 0.022, where a different verification batch shape changed floating-point reduction order and flipped the argmax. The distribution is preserved, but bit-for-bit reproduction on a GPU is not happening. If a regression suite pins exact model output, speculative decoding will break it, and that is not a bug you can fix. ### The speedup belongs to the workload, not the setup Deleting four words from one prompt (an instruction not to answer with an outline) changed what the model wrote, moved draft acceptance from 68% to 55%, and moved throughput 21%. Any single-prompt benchmark of this technique is one sample from a wide distribution. That’s why everything above uses eight tasks, and eight is still too few. ## What about just running a MoE model? The standard advice for bandwidth-limited hardware. A 90 GB DeepSeek-V4-Flash quant generates at 19.5–20.0 tok/s on the same tasks, 1.7× the dense baseline, but it **loses to DFlash2 on every task**, its prompt processing runs at roughly half the dense model’s rate (90–370 vs 206–753 tok/s), and it leaves 25 GB of headroom on the machine. The 2 GB drafter beats the 90 GB MoE on generation, prefill, and memory simultaneously. ## Where this is weak **DSpark ran from a community GGUF conversion**(`erlidev/Qwen3.8-27B-DSpark-Q8_0` ), unverified and untuned, so read “DFlash2 was faster here,” not “DFlash2 beats DSpark.”**Single-stream only.** Speculative decoding requires`--parallel 1` in llama.cpp; concurrent batching was not measured and may win for multi-user serving.**One model, one quantization.** Drafter quality is model-specific; none of these ratios should be assumed to transfer.**No quality evaluation** beyond the eight greedy-output comparisons, and the 273 GB/s bandwidth figure is the spec sheet, not a measurement.**DFlash2 is not merged.** It lives in[llama.cpp PR #27342](https://github.com/ggml-org/llama.cpp/pull/27342)and needs a local build; released containers ship the incompatible DFlash 1 (`wrong number of tensors; expected 81, got 58` ). ## Running it yourself ### Build & serve ``` git clone --depth 1 https://github.com/ggml-org/llama.cpp.git cd llama.cpp git fetch --depth 1 origin pull/27342/head:pr-dflash2 && git switch pr-dflash2 cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \ -DCMAKE_CUDA_ARCHITECTURES=121 -DLLAMA_CURL=OFF cmake --build build -j 20 --target llama-server ./build/bin/llama-server \ -m Qwen3.8-27B-UD-Q4_K_XL.gguf \ -md Qwen3.8-27B-DFlash2-Q8_0.gguf \ --spec-type draft-dflash \ -ngl 999 -ngld 999 -c 32768 --parallel 1 ``` The drafter is `incoai/Qwen3.8-27B-DFlash2-GGUF` on Hugging Face; its block size is baked in, so the `--spec-draft-*` flags have no effect on it (with a conventional draft model, `--spec-draft-p-min 0.75` mattered most). If you benchmark any of this: restart the server between measurements, use more than one prompt, and check whether your fastest number is physically possible before you believe it.