Trying "DFlash," a Diffusion-Model Approach to Parallel Draft-Token Generation, on Gemma

A developer benchmarked DFlash, a diffusion-model-based speculative decoding drafter from Z-Lab, against Gemma-4-12B-it's native Assistant MTP model on an RTX 3060 with 12GB VRAM using llama.cpp. DFlash failed to outperform the Assistant model on the JavaScript coding task, though the testing highlighted where the diffusion drafting approach is and isn't effective. DFlash, introduced in the paper "DFlash: Block Diffusion for Flash Speculative Decoding," predicts multiple tokens in parallel and claims broad model support and gains over EAGLE-3.

In the concept edition and the implementation/benchmark edition, we covered a speed-up technique for LLM generation called MTP Multi-Token Prediction . To recap briefly: a lightweight "draft model" predicts a handful of tokens ahead of time, and the main model checks them all at once. When the guesses are right, you leap ahead several tokens in a single step, which is what makes the whole thing feel faster. There's more than one way to build that draft model, and the one we're looking at this time, "DFlash," takes an unusual approach: it uses a diffusion model — the kind of technique you'd normally associate with image generation — to predict multiple tokens all at once instead of one at a time. It claims to support a wide range of models and to significantly outperform EAGLE-3, an existing approach. Those are the claims worth testing directly. The catch is that benchmarks like this are usually measured in an environment the vendor sets up. It's harder to find a case where someone ran it on their own GPU, against an opponent that already has a dedicated, well-optimized MTP model of its own — Gemma-4's native Assistant model. So this time, using the same setup as the implementation/benchmark edition an RTX 3060 with 12GB VRAM, llama.cpp, the same JavaScript coding task , we directly compared DFlash against Gemma-4-12B-it's Assistant model. The short version: DFlash did not outperform the Assistant model. The reasons are technically clear, though, and they draw a fairly clear picture of where DFlash is strong and where it isn't — that's what we'll dig into below. Two new token-prediction techniques appeared in quick succession: DSpark<sup 1</sup was released to speed up inference specifically for DeepSeek's own DeepSeek-V4, and it only supports DeepSeek-V4 / DeepSeek-V4-Flash. DFlash, on the other hand, supports a much wider range of models, each with its own dedicated DFlash model. Like Google's Assistant model, it's designed to be bolted on as an add-on to achieve token prediction. Z-Lab is a research group led by Zhijian Liu<sup 2</sup , an assistant professor at UC San Diego who is also a research scientist at NVIDIA. The lab works across the algorithm, systems, and application layers to make AI smaller, faster, and more efficient. This time, we wanted to understand how much of a speed-up DFlash actually delivers, verified through hands-on testing. DFlash is introduced in the paper "DFlash: Block Diffusion for Flash Speculative Decoding"<sup 3</sup . Speculative token prediction itself isn't new — the earliest paper on the idea was published by Google DeepMind in 2023, "Accelerating Large Language Model Decoding with Speculative Sampling"<sup 4</sup . Improvements continued quietly from there, culminating in 2025 in an MTP draft model called EAGLE-3. Even so, it apparently never escaped the autoregressive paradigm, and in the end didn't deliver a dramatic speed improvement. Meanwhile, diffusion models — the noise-removal mechanism commonly used in image generation — have been making their way into the LLM space. It started with Meta's LLaDa, and in Japan, KDDI's ELYZA Lab team released a model called "ELYZA-Diffusion-Instruct-1.0-Dream-7B"<sup 5</sup . Z-Lab's DFlash brings that diffusion-model property into the MTP draft model. The basic mechanism is shown below. php Input Token N -- Main Model -- h on ---------------------------------------- Predicted: N+1 | ^ v | KVCache | | | +--------------------------------------+--------------------------------+ | | DFlash Drafter v KV data injection | | | KVCache | | | | | | | Diffusion Model | | | | | | | Token Decoder | | | | | | | +---------v--------+ | | | | Predicted N+2 | | | | | + | | | | | Predicted N+3 | | | | | + | | | | | Predicted N+4 | | | | +---------+--------+ | | +--------------------------------------+--------------------------------+ | | | +--------------------------------------+---------------------------------------------+-------+ | Processing inside Main Model v | | | Uses causal attention to mask; | | | computes probs in parallel | | | +------------------------------+ | | | v v | | | Match found No match | | | Include matching portion Nothing included in output | | | - Predicted: N+2 | | | - Predicted: N+3 | | +------------------------------------------------------------------------------------+-------+ Figure 1: DFlash mechanism Looking at the mechanism, it resembles the Assistant model implemented in Gemma, but the big difference is in how the draft model itself predicts tokens. The latter half — the token-evaluation stage — closely mirrors Gemma's approach. With DFlash, the KV cache is built independently. As before, when the main model predicts token N+1, it uses that state to update its own KV cache. Data extracted from that update is then injected into DFlash's own KV cache, which is where DFlash's processing begins. Gemma's Assistant model runs this prediction step using a very small neural network, sequentially and at high speed, producing as many predicted tokens as needed before handing them off to the evaluation logic. DFlash, by contrast, doesn't use an autoregressive model inside its small neural network — it uses a diffusion model. Here, much like generating an image, it produces all of the needed predicted tokens, in the correct order, in one shot. The longer the maximum token length, the longer this takes, but it's still dramatically faster than doing it sequentially. What follows is the same as before: causal-attention-based masking runs in parallel, and the result determines which tokens are allowed to be output together. So the biggest contributor to any speed advantage comes down to the parts marked with blue and red boxes in the middle of the figure below. php For Gemma Assistant Input Token N -- Main Model -- Predicted: N+1 | +- Predicted: N+2 - seq - Predicted: N+3 - seq - Predicted: N+4 | v Verify -- Up to n OK -- Confirmed: N+2, N+3... For DFlash Input Token N -- Main Model -- Predicted: N+1 | | +-- Predicted: N+2 +---+-- Predicted: N+3 <-- Outputs all at once, order included | +-- Predicted: N+4 | | | v +--------- Verify -- Up to n OK -- Confirmed: N+2, N+3... Figure 2: Difference between Gemma's Assistant model and DFlash First, get the model files. hf download google/gemma-4-12B-it hf download z-lab/gemma4-12B-it-DFlash This time we're using llama.cpp build 9850. It's a good idea to grab the latest build. Use the Python conversion tool included with it to convert each model to GGUF format. Start with the main model. bash $ python convert hf to gguf.py \ ~/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/cxxxxxx...a/ \ --outtype bf16 --outfile gemma-4-12B-it-b16.gguf Next, convert the DFlash model to GGUF. The important thing here is --target-model-dir . When converting a DFlash model, it needs to be given a reference to the main model — that's the parameter it uses to build the converted DFlash output. bash $ python convert hf to gguf.py \ ~/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/cxxxxxx...a/ \ --outtype bf16 --outfile models/gemma-4-12B-it-DFlash-b16.gguf \ --target-model-dir \ ~/.cache/huggingface/hub/gemma-4-12B-it-qat-q4 0-unquantized/snapshots/c202...a/ If this fails with the message below, download tokenizer.model directly from the google/gemma-3-12b-it repository on Hugging Face and place it in the cache directory. INFO:hf-to-gguf:DFlash: Using tokenizer from target model: /home/aiuser/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/5854...7ced3 Traceback most recent call last : File "/data/user data/aiuser/llama.cpp/conversion/qwen.py", line 57, in set vocab self. set vocab sentencepiece File "/data/user data/aiuser/llama.cpp/conversion/base.py", line 1589, in set vocab sentencepiece tokens, scores, toktypes = self. create vocab sentencepiece File "/data/user data/aiuser/llama.cpp/conversion/base.py", line 1606, in create vocab sentencepiece raise FileNotFoundError f"File not found: {tokenizer path}" FileNotFoundError: File not found: /home/aiuser/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/585...ced3/tokenizer.model Quantizing these produces 4-bit models. $ /opt/llama/bin/llama-quantize \ models/gemma-4-12B-it-b16.gguf models/gemma-4-12B-it-q4 0.gguf q4 0 $ /opt/llama/bin/llama-quantize \ models/gemma-4-12B-it-DFlash-b16.gguf models/gemma-4-12B-it-DFlash-q4 0.gguf q4 0 At runtime, add the following arguments. Since the DFlash model acts as an add-on draft/MTP model, use --model-draft to point to it and enable MTP. /opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-q4 0.gguf \ --model-draft /opt/llama/models/gemma-4-12B-it-DFlash-q4 0.gguf \ -t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \ --device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8 0 -ctk q8 0 \ --no-warmup --no-cache-prompt --cache-ram 0 \ --spec-type draft-dflash --spec-draft-n-max 4 --spec-draft-device CUDA0 \ --chat-template-kwargs '{"enable thinking":true}' A quick rundown of the arguments: --model-draft <gguf file : -sm layer : --spec-type draft-dflash : --spec-draft-n-max 4 : --spec-draft-device CUDA0 : We ran the following simplified verification setup. Memory usage came out as follows. | Data category | Assistant CUDA0 | Assistant CPU | DFlash CUDA0 | DFlash CPU | |---|---|---|---|---| | Weight data | 6,390.19 | 540.00 | 6,637.69 | 787.50 | | KV cache data | 1,360.00 | — | 1,360.00 | — | | KV cache for Sliding Window | 765.00 | — | 765.00 | — | | Gated DeltaNet compute buffer | 533.80 | 180.80 | 533.80 | 180.80 | | MTP model weight data | 226.90 | 144.00 | 390.42 | — | | MTP token-to-piece cache size | — | — | 1.94 | — | | MTP model KV cache size | — | — | 640.00 | — | | MTP model Sliding Window KV cache size | — | — | 136.00 | — | | MTP model Gated DeltaNet compute buffer | 532.78 | 180.79 | 519.50 | 176.02 | | Total | 9,808.67 | 1,045.59 | 10,984.35 | 1,226.38 | Table 1: Memory usage breakdown by MTP method units: MiB . This run used text-only mode, so multimodal-model requirements are excluded. Of that, the memory used specifically by the MTP model itself was as follows — roughly double for DFlash. | | Assistant CUDA0 | Assistant CPU | DFlash CUDA0 | DFlash CPU | |---|---|---|---|---| | MTP model memory usage | 759.68 | 324.79 | 1,687.86 | 176.02 | Table 2: Total memory used by the MTP model, by MTP method units: MiB The main driver here is how the two approaches use the KV cache. The Assistant model shares its KV cache with the main Gemma model. DFlash, by contrast, keeps its own independent KV cache — meaning it has to hold the same cache structure as the main Gemma model a second time. That's the biggest factor behind the difference. As in the previous verification, we compared token ingestion speed turn by turn. There was no major throughput difference between the Assistant model and DFlash for reading tokens in. | Turn | Assistant tok/s | DFlash tok/s | |---|---|---| | 1 | 138.34 | 53.72 | | 2 | 710.71 | 652.47 | | 3 | 733.53 | 739.52 | | 4 | 735.43 | 708.22 | Figure 3: Token read-in speed by turn For output speed per turn, the Assistant model was clearly ahead — DFlash consistently trailed by about 10–20 tokens per second. | Turn | Assistant tok/s | DFlash tok/s | |---|---|---| | 1 | 62.80 | 39.37 | | 2 | 62.04 | 48.42 | | 3 | 66.68 | 52.40 | | 4 | 62.77 | 48.73 | Figure 4: Token output speed by turn Here is the raw measurement data behind the numbers above. Note that time is in milliseconds. Reading | Turn | Assistant time ms | Assistant tokens | Assistant tps | DFlash time ms | DFlash tokens | DFlash tps | |---|---|---|---|---|---|---| | 1 | 181 | 25 | 138.34 | 577 | 31 | 53.72 | | 2 | 3,004 | 2,135 | 710.71 | 3,374 | 2,202 | 652.47 | | 3 | 6,361 | 4,666 | 733.53 | 6,391 | 4,727 | 739.52 | | 4 | 9,559 | 7,030 | 735.43 | 10,173 | 7,205 | 708.22 | Generation | Turn | Assistant time ms | Assistant tokens | Assistant tps | DFlash time ms | DFlash tokens | DFlash tps | |---|---|---|---|---|---|---| | 1 | 33,186 | 2,084 | 62.80 | 54,413 | 2,142 | 39.37 | | 2 | 40,281 | 2,499 | 62.04 | 51,096 | 2,474 | 48.42 | | 3 | 35,046 | 2,337 | 66.68 | 46,701 | 2,447 | 52.40 | | 4 | 45,532 | 2,858 | 62.77 | 55,774 | 2,718 | 48.73 | Table 3: Raw measurements from the logs Overall, inference with DFlash trailed the Assistant model by under 20 seconds. The Assistant model was already a fairly well-optimized setup going in, which may be part of why the diffusion model's theoretical advantage didn't stand out here. As before, output speed increased somewhat over the course of each turn for both the Assistant model and DFlash, but the Assistant model was faster throughout. We're omitting the detailed per-token scatter plots for each model here, since the turn-by-turn averages above and the raw log data in Table 3 already capture the trend. In a few cases we also observed DFlash's output speed peaking around 1,700 tokens and then dropping off slightly after that. As in the previous verification, let's also look at the token acceptance rate. | Turn | Acceptance | Accepted | Generated | |---|---|---|---| | 1 | 41.03% | 1,331 | 3,244 | | 2 | 53.36% | 1,684 | 3,156 | | 3 | 62.02% | 1,744 | 2,812 | | 4 | 57.26% | 1,892 | 3,304 | Table 4: DFlash token acceptance rate Comparing this against the Assistant model: | | Assistant Acceptance | Assistant Accepted | Assistant Generated | DFlash Acceptance | DFlash Accepted | DFlash Generated | |---|---|---|---|---|---|---| | Max | 88.27% | 2,196 | 2,648 | 62.02% | 1,892 | 3,304 | | Min | 75.34% | 1,564 | 2,064 | 41.03% | 1,331 | 2,812 | | Avg | 81.06% | 1,868 | 2,305 | 53.42% | 1,663 | 3,129 | Table 5: Acceptance rate comparison — Assistant left vs. DFlash right This suggests that DFlash's underwhelming result comes down to its lower acceptance rate. If model tuning progresses further and a faster-processing diffusion model becomes possible, DFlash might eventually surpass the Assistant model. Through this verification, we confirmed that DFlash falls a bit short of the Assistant model. But maybe tweaking the parameters would speed things up? With that in mind, we tried the following. The predicted-token count is currently set to 4. That's the same value DFlash used in the previous Assistant-model verification, kept for comparability. Raising this significantly might help — so we bumped --spec-draft-n-max from 4 to 15. /opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-q4 0.gguf \ --model-draft /opt/llama/models/gemma-4-12B-it-DFlash-q4 0.gguf \ -t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \ --device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8 0 -ctk q8 0 \ --no-warmup --no-cache-prompt --cache-ram 0 \ --spec-type draft-dflash --spec-draft-n-max 15 --spec-draft-device CUDA0 \ --chat-template-kwargs '{"enable thinking":true}' Here's what came out of running it: prompt eval time = 122.83 ms / 31 tokens 3.96 ms per token, 252.39 tokens per second eval time = 48833.63 ms / 1893 tokens 25.80 ms per token, 38.76 tokens per second total time = 48956.46 ms / 1924 tokens graphs reused = 1444 draft acceptance = 0.10901 1174 accepted / 10770 generated , mean len = 2.64 The complete opposite of what we hoped for: acceptance dropped, and speed dropped along with it. It generated a lot more candidate tokens, but almost all of them were rejected — simply lengthening the prediction window clearly wasn't the answer. Here's the breakdown by turn: | Turn | Acceptance | Accepted | Generated | |---|---|---|---| | 1 | 10.90% | 1,174 | 10,770 | | 2 | 13.59% | 2,059 | 15,150 | | 3 | 19.54% | 2,090 | 10,695 | | 4 | 19.30% | 2,310 | 11,970 | | | Acceptance | Accepted Tokens | Generated Tokens | |---|---|---|---| | Max | 19.54% | 2,310 | 15,150 | | Min | 10.90% | 1,174 | 10,695 | | Avg | 15.83% | 1,908 | 12,146 | The mean len=2.64 value in the log above appears to represent roughly "how many tokens tend to get accepted." Looking at the per-turn breakdown above, the average was 15.83%<sup 6</sup , suggesting that --spec-draft-n-max=4 was in fact the right setting for this model. What if we raised the model's own output precision instead? We tried rebuilding DFlash on that basis. For this run, since we'd need to manually apply the same quantization level, we couldn't use an Unsloth Dynamic 2.0–quantized model the way we did for the Assistant model. Instead, we rebuilt using Gemma-4-12B-it-QAT including the DFlash model , hoping it might improve performance. The results, unfortunately, were not what we expected. Output was so slow that we stopped measuring after the second turn. prompt eval time = 475.69 ms / 31 tokens 15.34 ms per token, 65.17 tokens per second eval time = 102682.33 ms / 1963 tokens 52.31 ms per token, 19.12 tokens per second total time = 103158.02 ms / 1994 tokens graphs reused = 1953 draft acceptance = 0.00025 2 accepted / 7844 generated , mean len = 1.00 This configuration is not viable: throughput dropped further, and the acceptance rate fell to 0.00025, or 0.25% — the lowest we observed in this test. With mean len=1.00 , essentially every generated token was rejected, a clear indication that DFlash should not be used in this configuration. It looks like the DFlash model Z-Lab provided is really only usable with the vanilla Gemma-4-12B-it model. That's unfortunate — if that's the case, the Assistant model, which comes with a properly matched MTP model for each release, seems like the more practical choice. The likely cause here is that Gemma-4-Assistant's draft model is simply fast enough that it processed tokens more quickly than DFlash's draft model. As covered earlier, the speed advantage a diffusion-based drafter like DFlash offers over a conventional MTP draft model comes from being able to output candidate tokens "all at once." Against that, Gemma-Assistant has the following working strongly in its favor, which appears to have canceled out DFlash's advantage: First, Gemma-Assistant comes with KV-cache sharing built in from the start, while DFlash keeps an independent KV cache that has to be injected fresh every time. That's an advantage for Gemma-Assistant. On top of that, Gemma-Assistant's hidden dimension is only 1,024, with a 4-layer structure. DFlash's hidden dimension, as noted at the end of this article, matches the main model at 3,840, with 5 layers — meaning its compute cost is far heavier than Gemma-Assistant's. However good DFlash's diffusion model is at generating tokens in a single batch, more compute per step still means more time spent per step. In this case, we should conclude that Gemma-Assistant simply already had a more optimized setup, and DFlash wasn't able to get ahead of it. It's also worth remembering that DFlash's original benchmark comparison was against EAGLE-3, not Gemma-Assistant — so it's possible Gemma-Assistant is simply a stronger comparison baseline than the one DFlash was originally built to outperform. That said, while the established option produced better results this time, it's possible a pairing with, say, Qwen3.5 would have produced a more favorable result. This time we introduced DFlash, a technique from the UC San Diego UCSD research team Z-Lab that takes the "MTP" technology covered previously and pushes it in a new direction. Applying a diffusion model to a draft model was a genuinely ambitious idea, but in this test it wasn't able to beat Gemma's native draft model, Assistant. Tracing the cause, the token acceptance rate was lower than the native draft model's, and that penalty appears to have been the dominant factor. Without a way to understand how to raise that acceptance rate, it's hard to say at this point whether further speed gains are realistic. We also think the matchup itself didn't help. Gemma-4's Assistant model is built with practicality in mind and makes full use of Gemma-4's shared KV-cache mechanism, whereas DFlash, as before, has to write to an independent KV cache — and that overhead seems to have tipped the balance toward lower throughput. On the broader question of putting diffusion models to work in LLMs, LLaDa and Dream are the well-known names, but more recently Google itself has released a model called Diffusion Gemma. We're currently in the middle of testing it ourselves, and the throughput numbers so far are notably high. We plan to cover those results in a dedicated article. As Coding Agent usage keeps climbing, LLM throughput has become a common pain point, and rising VRAM costs have only sharpened the demand for models that can deliver solid throughput even on unified memory. This kind of research is very much a "fail your way forward" field, and new techniques are emerging as we speak. We'll keep watching this space and try to keep up. DFlash: Block Diffusion for Flash Speculative Decoding Jian Chen, Yesheng Liang, Zhijian Liu https://arxiv.org/pdf/2602.06036 https://arxiv.org/pdf/2602.06036 Accelerating Large Language Model Decoding with Speculative Sampling DeepMind: Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper https://arxiv.org/pdf/2302.01318 https://arxiv.org/pdf/2302.01318 DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence DeepSeek-AI https://arxiv.org/pdf/2606.19348 https://arxiv.org/pdf/2606.19348 Speculative decoding feat: add DFlash support https://github.com/ggml-org/llama.cpp/pull/22105 https://github.com/ggml-org/llama.cpp/pull/22105 We've omitted the architecture diagram here to keep the English edition concise — the description below covers the key points. The DFlash model, which adds MTP capability to Gemma-4-12B-it, is architecturally distinct from Gemma-4-Assistant: rather than sharing a KV cache the way Assistant does, it's a fully separate model. In terms of its activation function and related choices, it's actually closer to a Qwen-style model. Its LA/GA notation follows Gemma-4's own convention — LA meaning Sliding Window Attention and GA meaning Full Attention. During prediction, state vectors are pulled from layers 1, 10, 19, 27, 36, and 45, concatenated, and normalized. That combined information is then injected into the draft side's KV cache. The input to the draft model is a query built by combining the embedding vectors of the N confirmed tokens with as many MASK tokens as the number of predictions needed. Unlike the Assistant model, this approach uses non-causal attention, and internally relies on Masked Diffusion — that's the key structural difference. This mechanism lets DFlash derive all of its predicted tokens at once; from there, the rest of the pipeline follows the same logic as the Assistant model, outputting whichever tokens get accepted. EAGLE3 is version 3 of the EAGLE Extrapolation Algorithm for Greater Language Model Efficiency series, documented in the following papers: Development has been led by Yuhui Li of Peking University, first author on these papers, together with a joint research team spanning Peking University, Microsoft Research, the University of Waterloo, and the Vector Institute. In the open-source community, they operate under the name SafeAILab. Successor models are also in development: SafeAILab released EAGLE3.1 in May 2026, and a team at AWS AI Labs has developed P-EAGLE Parallel-Drafting EAGLE , released in February 2026. Both treat the sequential bottleneck of causal attention as the core problem, and both are focused on how to parallelize that part of the pipeline further. This article is an English adaptation of the original Japanese post published on Zenn: "ドラフトトークンを並列生成する拡散モデル方式「DFlash」をGemmaで試した" https://zenn.dev/highreso/articles/ab3dbc20ce58ff , by Yuichi Tominaga. https://arxiv.org/pdf/2606.19348 https://arxiv.org/pdf/2606.19348 ↩ https://zhijianliu.com/ https://zhijianliu.com/ , https://z-lab.ai/ https://z-lab.ai/ ↩ https://arxiv.org/pdf/2602.06036 https://arxiv.org/pdf/2602.06036 ↩ https://arxiv.org/pdf/2302.01318 https://arxiv.org/pdf/2302.01318 ↩ https://huggingface.co/elyza/ELYZA-Diffusion-Instruct-1.0-Dream-7B https://huggingface.co/elyza/ELYZA-Diffusion-Instruct-1.0-Dream-7B ↩ Average acceptance rate across the four turns shown in the table below. ↩