DFlash changes what tokens per second means

A configuration using the DFlash speculative decoding drafter with Meta Muse Glimmer 30B on an NVIDIA RTX PRO 4000 Blackwell SFF GPU reached 84.64 tokens per second on a coding task, a 4.50 times speedup over regular decoding's 17.98 tok/s, but dropped to 38.34 tok/s on mixed workloads and 21.56 tok/s with a full 262,116-token KV cache. The results show that tokens per second now measures predictability as much as hardware, with code being unusually friendly to speculative decoding due to its structured nature.

I spent a night trying to fit a dense 30B model, 256K context, vision, and speculative decoding onto one 24 GB GPU. The fastest quant lost. The quant with the lowest perplexity lost too. What won was the configuration that made the whole system useful, not any single number impressive. The final setup runs Meta Muse Glimmer 30B on an NVIDIA RTX PRO 4000 Blackwell SFF capped at 70 watts. It holds the target model, a separate vision projector, a five-layer DFlash drafter, and a 262,144-token slot on one card. On a coding task it reaches 84.64 tokens per second. On mixed code, prose, reasoning, and infrastructure work it falls to 38.34. With the KV cache actually filled to 262,116 input tokens, decode falls again to 21.56. Same weights. Same GPU. Same drafter. Three very different machines, depending on what they are asked to produce and where the cursor sits in memory. With speculative decoding, tokens per second is no longer purely a hardware benchmark. It is also a predictability benchmark. That is the useful result. DFlash did not just make this model faster. It changed what throughput measures. DFlash is not a faster Glimmer Muse Glimmer 30B https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF is a dense model with roughly 29.6 billion parameters, 52 layers, and a native 131,072-token context that can be extended to 262,144. Dense matters here. Every generated target token activates the whole model. There is no MoE shortcut where only a small subset of parameters runs. DFlash sits beside it. The drafter has five layers and prepares a block of candidate tokens in parallel. In this llama.cpp configuration, n max=15 means up to 15 draft candidates around a 16-token block. Glimmer then verifies that block with the full target model. Candidates survive only while they match the target's accepted continuation. The first mismatch ends the accepted prefix. The drafter is not blind autocomplete. It receives target-model features from layers 1, 13, 25, 37, and 49, injecting information about Glimmer's internal state into its own cache before proposing the next block. That is the key idea in the DFlash paper https://arxiv.org/abs/2602.06036 : a lightweight block-diffusion model drafts in parallel, while the expensive autoregressive model verifies. DFlash does not make one Glimmer forward pass cheaper. It tries to buy several output tokens with that pass. When it guesses well, the cost of verification is amortized across an accepted prefix. When it guesses badly, the drafter and verification work become overhead. This is also why speculative decoding can be lossless with respect to the target model under the same sampler. The drafter proposes. The target remains the authority. A weak drafter should reduce speed, not intelligence. If output quality changes materially, suspect quantization, sampling differences, or an implementation bug before blaming the speculative idea itself. Code and planning are different workloads On the same Q5 K M target, regular decoding produced 17.98 tok/s. DFlash on the long coding task produced 80.87 tok/s, a 4.50 times speedup. The best tuned 256K run reached 84.64 tok/s with 38.67% token acceptance. Then I ran a mixed workload: code, prose, reasoning, and infrastructure. Throughput dropped to 38.34 tok/s and global acceptance to 14.41%. | Workload | Decode | Acceptance | What it measures | |---|---|---|---| | Regular decoding, code | 17.98 tok/s | n/a | Target model alone | | DFlash, code | 80.87 tok/s | 36.18% | Predictable structured output | | DFlash, best tuned code | 84.64 tok/s | 38.67% | 256K slot, vision loaded | | DFlash, mixed agent work | 38.34 tok/s | 14.41% | Code, prose, planning, infra | | DFlash, full KV cache | 21.56 tok/s | workload-specific | 262,116 input tokens | Code is unusually friendly to speculative decoding. Syntax, indentation, APIs, repeated identifiers, boilerplate, and local patterns constrain the next tokens. After for int i = 0; i < , there are relatively few sensible continuations. A drafter can often travel several tokens before it diverges. Planning is different. After “the safest migration strategy is”, several continuations can be equally correct. The drafter chooses one. Glimmer chooses another. Neither continuation has to be bad, but an exact-token verifier sees a mismatch, rejects the remaining speculative prefix, and starts another cycle. This means an agent has at least two throughput regimes. During implementation, repetitive edits, code completion, JSON, command lines, and schema-constrained tool calls can fly. During architecture, ambiguous reasoning, or conversational explanation, DFlash may spend much more time asking the target, “did you mean this exact path?” A single average hides that difference. If an agent benchmark contains 80% code emission, DFlash looks extraordinary. If it contains long planning traces and divergent prose, the same setup looks merely decent. Neither result is false. The workload distribution is part of the benchmark. The acceptance-rate trap The obvious metric is acceptance rate: what percentage of proposed draft tokens survived verification. It is useful, but by itself it can point to the wrong configuration. | Maximum draft | Acceptance | Decode | |---|---|---| | 4 tokens | 47.46% | 34.50 tok/s | | 8 tokens | 27.00% | 40.54 tok/s | | 12 tokens | 20.33% | 42.22 tok/s | | 15 tokens | 17.99% | 47.20 tok/s | Draft 4 won acceptance and lost throughput. Draft 15 accepted a much smaller percentage yet generated 36.8% faster. A longer block proposed more wrong tokens, but the accepted prefix amortized each expensive target verification more effectively. The better companion metric is average acceptance length, usually written as τ : how many consecutive speculative tokens survive per target pass. On my coding run, τ reached 6.39. That is closer to the economic question. How many output tokens did one expensive verification buy? Even τ is not enough alone. Drafter latency, verification cost /glossary/verification-cost , batch shape, backend synchronization, and KV-cache position all influence the result. But τ explains the mechanism much better than a raw acceptance percentage. The CPU was quietly putting a handbrake on the GPU The most important performance fix was not a new quant. It was removing a boundary crossing. The drafter ran on the GPU, but greedy token selection still passed through a CPU-side path. That forced device-to-host work and synchronization inside a loop whose entire value comes from being cheap and parallel. The GPU could finish its matrix work and then wait while the host selected draft IDs. The llama.cpp GPU-argmax change https://github.com/ggml-org/llama.cpp/pull/26842 adds ggml argmax directly to the drafter's backend graph. My debug trace showed the graph grow from 179 to 180 nodes when DFlash was enabled. The selected draft token stayed in the backend path instead of making the CPU the metronome. On the mixed five-by-256-token series, that isolated change moved throughput from 36.30 to 38.34 tok/s, a 5.6% gain. That sounds modest until the workload becomes predictable enough for DFlash to compound the saving. In the final coding setup, the target alone managed 17.98 tok/s while DFlash with GPU argmax reached 80.87 tok/s. The lesson is broader than this model. A speculative decoder is a pipeline, not a checkbox. The target can be perfectly optimized while a tiny host-side operation serializes the draft loop. If utilization, acceptance, and wall-clock throughput do not agree, inspect graph boundaries and synchronization before buying a larger GPU. Two integration bugs made NVFP4 look worse than it was NVFP4 initially produced broken output. The easy conclusion would have been that Blackwell FP4 was immature or that the checkpoint was bad. Tensor comparisons told a different story: sampled weights correlated at roughly 0.993, while embeddings and the language-model head were even closer. The packed weights were not the primary problem. I found two independent integration faults. The generic compressed-tensors NVFP4 conversion path skipped Muse-specific Q/K RoPE permutation and construction of constant Q/K norms. A model can load successfully and still be semantically wrong if architecture-specific tensor transformations are omitted. The Muse runtime loaded Q, K, V, and output projection used their scales, while the attention gate and the FFN up, gate, and down matrices did not. Nine changed lines restored those four scale paths and stopped activation values from exploding. .scale tensors but did not pass all of them into the graph. After the fixes, NVFP4 generated clean text and became the throughput leader in one hybrid configuration. That matters because “the model runs” is a weak test for a new quant format. You need semantic output, activation sanity, tensor-by-tensor conversion checks, perplexity, and a representative generation workload. A runtime bug can masquerade as bad quantization. A conversion bug can masquerade as a bad model. A sampling default can masquerade as a slow drafter. Inference engineering is mostly refusing the first plausible explanation. The fastest quant still lost The experimental hybrid combined native NVFP4 with Q4 attention treatment and a higher-precision head. It reached 94.36 tok/s on code with 38.69% acceptance. That was the fastest result of the night. It also produced worse WikiText-2 perplexity: 5.6493 versus 5.5420 for Q5 K M. The speed gain was real. So was the quality loss. For a production coding agent, the correct decision was to reject it. | Target | WikiText-2 PPL | Code TPS | Acceptance | Decision | |---|---|---|---|---| | Q5 K M | 5.5420 ± 0.1179 | 80.87 | 36.18% | Selected | | Meta kquant, 17 GB | 5.5620 ± 0.1182 | 83.92 | 34.68% | Speed alternative | | Q4 | 5.5797 | 82.00 | 31.77% | Good density | | Hybrid NVFP4/Q4 | 5.6493 | 94.36 | 38.69% | Rejected on quality | | NVFP4, BF16 head | 5.6937 | 69.10 | 26.41% | Rejected | The opposite extreme lost too. Q5 K XL improved perplexity from 5.5420 to 5.5278, only 0.26%. That difference was far smaller than the test's ±0.1179 uncertainty. Yet on the mixed workload it fell from 38.34 to 33.50 tok/s, roughly 13% slower, with worse DFlash acceptance. So the perplexity winner bought no statistically persuasive quality improvement and paid a double-digit throughput penalty. Q5 K M sat on the Pareto frontier: better measured quality than the smaller quants, much better DFlash compatibility than Q5 K XL, and enough VRAM left for 256K context plus vision. Why the quant label is not the model “Q4 versus Q5” is too crude for modern inference work. A model is not one homogeneous block of equally sensitive numbers. Embeddings, attention projections, gates, feed-forward tensors, norms, and lm head tolerate quantization differently. Importance matrices, or imatrix calibration, measure which weights matter under representative activations. Mixed-quant recipes then spend bits where error hurts and remove them where it does not. Unsloth's dynamic GGUF variants, K-quants, and hybrid NVFP4 experiments all exploit versions of this idea: the file's average bits per weight tells you its size, not how intelligently those bits were allocated. Blackwell-native NVFP4 adds another dimension. Hardware support can make 4-bit arithmetic extremely fast, but format support does not erase architectural sensitivity. Leaving large embeddings or lm head in BF16 can preserve quality, but consumes several gigabytes. Quantizing them saves VRAM, but may damage logits disproportionately. Treating attention and FFN identically may waste quality or speed. The best recipe is tensor-aware, architecture-aware, and measured with the actual drafter. DFlash makes this coupling tighter. Two target quants with nearly identical perplexity can expose different hidden features to the drafter and produce different acceptance lengths. Quant quality and speculative compatibility are separate axes. Perplexity alone cannot select the winner. One hidden sampling default cost 8% Meta recommends temperature 1.0, top-p 0.95, and top-k 64 for Muse Glimmer. I set all three. The server still applied llama.cpp's default min-p=0.05 . Removing that unrequested fourth filter with min-p=0 raised the final 256K vision-enabled result from 78.29 to 84.64 tok/s. Acceptance rose from 36.18% to 38.67%. Same model. Same quant. Same GPU. An 8.1% gain from making the sampler match the model's documented defaults. Sampling changes which candidate becomes authoritative, so it also changes whether the drafter's prefix survives. A benchmark that reports temperature but omits top-p, top-k, min-p, and draft sampling is not reproducible. A 256K context is not tested until it is full A server accepting --ctx-size 262144 proves almost nothing. KV memory may be reserved lazily, CUDA work buffers can grow during prefill, and long-cache attention changes decode cost. “It loaded” is not a stress test. I filled it. Input: 262,116 tokens, followed by output. Result: no out-of-memory failure. Prompt processing: 371.24 tok/s. Far-cache decode: 21.56 tok/s. VRAM: 22,920 of 24,467 MiB, or 93.68%. Remaining VRAM: 1,547 MiB. The separate quantized vision projector also remained loaded and a real image request passed. The final short-code run at 256K was not slower than the equivalent 128K configuration because an empty larger slot is cheap with Q8 KV. The far end was different: once the cache was actually occupied, decode fell to 21.56 tok/s. This is the long-context tax that load-only tables hide. Context capacity and context performance are not the same claim. The final configuration The winning profile was not the fastest component in every column. It was the best complete system: Target: Muse Glimmer 30B UD-Q5 K M. Context: 262,144 tokens. Vision: quantized Muse mmproj loaded on the same GPU. Speculation: dflash-kquant , n max=15 . KV cache: Q8 for the target, F16 for the much smaller drafter. Execution: GPU argmax, CUDA graph optimization, batch/ubatch 1024/256, target and draft threads 8/8. Sampling: temperature 1.0, top-p 0.95, top-k 64, min-p 0. Measured short-code result: 84.64 tok/s, 38.67% acceptance. Q8 target KV was a better trade than F16. It saved about 740 MiB and improved throughput in this setup. Keeping the drafter KV in F16 cost only about 32 MiB over Q4 and performed slightly better. The small cache should not be optimized with the same aggression as the giant one. This is the same principle I use in routing by task instead of vendor /blog/route-by-task-not-vendor-open-weight-ai-architecture : optimize the system around the expensive constraint, not every component symmetrically. Where DFlash shines, and where it dies DFlash is excellent when output has a narrow continuation: - code completion and repetitive implementation; - structured JSON and schema-constrained outputs; - boilerplate, test generation, migrations, and predictable refactors; - command sequences and tool calls with stable syntax; - domains where the drafter's target features strongly constrain the next block. It weakens when many continuations are valid: - architecture planning and open-ended reasoning; - creative prose and conversational answers; - high-temperature generation; - frequent topic shifts or retrieval inserts; - very long occupied KV caches, where every target verification itself becomes more expensive. There are operational costs too. The drafter consumes model memory and its own KV cache. Vision consumes more. Longer draft blocks can waste compute. Backend support has to keep the loop on-device. The target and drafter must agree on vocabulary, architecture hooks, and sampling behavior. A 1.6 GB sidecar that gives 4.5 times on code is a bargain. The same sidecar giving a marginal gain on divergent chat may not be. The right question is not “does DFlash work?” It is “what fraction of my production output is predictable enough for DFlash to amortize target passes?” What a useful speculative-decoding benchmark must report A single tok/s number is now under-specified. At minimum, report: Target model and exact quant recipe. Include mixed tensors, imatrix use, precision exceptions, and file size. Drafter and block length. “DFlash enabled” says too little. Task distribution. Separate code, planning, chat, tool calls, and mixed agent work. Acceptance rate and average acceptance length τ. One without the other can mislead. All sampling parameters. Include hidden defaults such as min-p. Prefill and decode separately. Do not blend prompt ingestion with generation. KV-cache position. Empty 256K and occupied 256K are different workloads. VRAM after real fill. Load-time allocation is not enough. Backend path. State whether draft token selection stays on GPU. Quality beside speed. At least perplexity plus a representative task eval. This is predictability-adjusted throughput: effective generation speed measured over a declared output distribution, draft configuration, sampler, quant, and KV-cache position. It is less convenient than one heroic number. It is also much closer to production reality. Key takeaways - DFlash does not accelerate a Glimmer forward pass. It reduces how often the target must run by getting several verified tokens from one pass. - On the same GPU and weights, code reached 84.64 tok/s while mixed agent work reached 38.34. Output predictability is part of throughput. - Acceptance percentage alone selected the wrong draft length. Draft 15 was faster than draft 4 despite much lower acceptance. - Moving drafter argmax from CPU into the GPU graph removed a synchronization boundary from the hottest loop. - The fastest NVFP4 hybrid lost because its perplexity was worse. Q5 K XL lost because a statistically weak 0.26% PPL gain cost roughly 13% throughput. - Quant labels hide tensor-level decisions. Imatrix calibration, precision exceptions, architecture transforms, and DFlash compatibility all matter. - A hidden min-p default cost 8.1%. Sampling is part of the performance configuration. - The 256K setup survived a real 262,116-token fill at 93.68% VRAM, but far-cache decode dropped to 21.56 tok/s. Next time someone posts a tok/s figure, ask what the model generated, how many draft tokens survived each target pass, whether draft selection stayed on the GPU, and where the cursor sat in the KV cache. Without those facts, it is not a benchmark. It is a screenshot.