Inside LLaDA-8B on a real GPU: out-of-order decoding, the “San Angeles” crash, and why naive bidirectional attention runs 2.75x slower without a KV cache
I ran the open-weight diffusion language model GSAI-ML/LLaDA-8B-Instruct against a matched autoregressive twin (Qwen/Qwen2.5-7B-Instruct) and FacebookAI/roberta-large on an NVIDIA A100-SXM4-40GB GPU, and compared the published configs of Dream-7B, Fast-dLLM v2 and LLaDA-MoE. Unmasking all tokens in a single step crashes into two-word mode collisions ("San Angeles": a 47.45% chance when both slots are sampled at once, computed from the model's own probabilities) and 54.10% word stutter because softmax heads predict independent marginals rather than joint distributions. Worse, naive 64-step bidirectional diffusion without a KV cache drops to 9.77 tokens per second on a 1,024-token prompt—2.75x slower than eager autoregressive decoding (26.87 tok/s). Block-cached prefix reuse plus 0.90 confidence thresholding compresses generation into 10 forward passes; built from measured prefill and block-pass timings, that gives an estimated 129.07 tokens per second on 1,024-token prompts (148.13 tok/s at 256 prompt tokens).
When I first loaded GSAI-ML/LLaDA-8B-Instruct onto an NVIDIA A100-SXM4-40GB GPU and watched it write a 64-token Python binary_search function in 16 parallel denoising steps, the very first forward pass broke twenty years of autoregressive intuition.
In step 1 (one forward pass, about 42 ms once the GPU is warm), the model did not write def at position 0. Instead, out of 64 blank [MASK] slots (mask_token_id = 126336), it locked four non-adjacent positions simultaneously: position 1 (python), position 6 ((arr), position 8 (target), and position 63 (<EOS>). At step 2, it filled in position 0 (the opening triple-backtick code fence), position 3 (def), position 4 (binary), and position 7 (,). At step 4, it locked the final period (.) at position 61—anchoring the end of the prose explanation while positions 15 through 60 were still completely blank masks.
Then I ran two experiments that exposed the real engineering trade-offs behind the viral “10x faster parallel AI” headlines:
First, when I forced LLaDA-8B to unmask all tokens in 1 single step (S = 1), its own probabilities gave a 47.45% chance of splicing competing answers into chimeric hybrids such as "San Angeles" or "Los Francisco" when both slots are sampled at once—and collapsed into 54.10% adjacent word stutter on prose.
Second, when I timed vanilla uncached LLaDA-8B (S = 64) against its matched autoregressive twin Qwen/Qwen2.5-7B-Instruct on a 1,024-token prompt, naive diffusion was not faster. Because bidirectional attention invalidates the key-value cache on every step and shifts the 1,088-token pass into the compute-bound regime (17.4 TFLOPs per step), uncached 64-step LLaDA-8B ran at 9.77 tokens per second—2.75x slower than KV-cached Qwen2.5-7B at 26.87 tokens per second.
Here is everything I measured inside the tensors on our A100 GPU, how modern 2025–2026 diffusion language models actually work under the hood, and how confidence thresholding plus block KV caching turns that long-prompt slowdown into an estimated 4.80x–5.47x speedup over autoregressive decoding.
- Why Autoregressive Decoding Starves Your GPU — and Why 15% BERT Cannot Generate
To see why researchers are replacing the causal decoder mask, start with the hardware arithmetic of Qwen/Qwen2.5-7B-Instruct (7,615,616,512 parameters in bfloat16, occupying 15.23 GB of A100-40GB HBM2 memory at 1,555 GB/s peak bandwidth).
During autoregressive generation, causal masking (M[i, j] = -inf for j > i) ensures that token i only attends to past tokens 1..i. That gives us an exact key-value (KV) cache, so generating 1 new token only computes queries, keys, values, and feed-forward activations for a single column ([1, 1, 3584]). However, to multiply that single vector, the A100 still has to stream all 15.23 GB of model weights out of HBM2 memory on every single token. At batch size 1, arithmetic intensity sits near 1 FLOP per byte (far below the A100's ~200 FLOP/byte roofline balance point)—leaving the 312 TFLOPS of BF16 Tensor Cores more than 95% idle. Even a fully compiled kernel is hardware-capped near ~100 tokens per second by HBM2 bandwidth (1,555 GB/s / 15.23 GB), and in standard Hugging Face PyTorch eager execution (model.generate at batch size 1), per-layer CUDA kernel launch overhead plus HBM weight reads cap single-stream generation at 26.87–27.24 tokens per second across 64- to 1,024-token prompts (27.06 tok/s at 256 tokens is about 412 GB/s of effective weight bandwidth).
If we remove the causal mask (is_causal = False) so the transformer processes 64 token positions in parallel per weight read, why couldn't we use classic masked language models like BERT or RoBERTa-large to generate paragraphs years ago?
I tested FacebookAI/roberta-large (355M parameters, trained with a fixed 15% MLM objective) against GSAI-ML/LLaDA-8B-Instruct (8.016B parameters, trained with a continuous t ~ U(0, 1] objective) across six corruption ratios t from 0.15 (15% masked) up to 1.00 (100% masked) across four technical benchmark sentences on the A100 (seed 42). Entropy is measured on the first forward pass for both models; RoBERTa fills all masks in a single pass, while LLaDA fills them over 8 iterative unmasking steps, which is how each model is meant to be used. While the two checkpoints differ in parameter scale, their behavior across t illustrates the structural difference between fixed-ratio MLM pre-training and continuous-time diffusion pre-training:
At t = 0.15 (1/t = 6.67): RoBERTa-large recovers 33.33% of masked tokens cleanly with 4.138 bits of predictive entropy and 0.0% adjacent token repetition, while LLaDA-8B recovers 45.83% with 2.413 bits of entropy and 0.0% repetition.
At t = 0.30 (1/t = 3.33): RoBERTa-large reaches 44.20% accuracy (4.097 bits entropy, 1.85% repetition), and LLaDA-8B reaches 50.45% accuracy (4.369 bits entropy, 0.0% repetition).
At t = 0.50 (1/t = 2.00): RoBERTa-large drops to 13.99% accuracy as entropy climbs to 6.818 bits, while LLaDA-8B achieves 21.19% accuracy at 5.200 bits of entropy.
At t = 0.75 (1/t = 1.33): RoBERTa-large entropy surges to 9.271 bits and adjacent word repetition jumps to 23.19%, versus 6.142 bits and 7.58% iterative repetition for LLaDA-8B.
At t = 0.90 (1/t = 1.11): Seeing a 90% masked canvas pushes fixed-15% RoBERTa-large far outside its training distribution: its entropy spikes to 11.477 bits, top-1 confidence drops to 5.14%, and 54.73% of its single-pass output collapses into repeated adjacent tokens. (At t = 1.00 both models break down in this simple test, with 51.84% and 45.63% repetition: LLaDA gets no chat template here and only 8 steps for the whole sentence, so full generation needs the longer remasking schedule shown in Section 2.) Across iterative denoising from t = 0.90, LLaDA-8B holds step-1 entropy at 7.875 bits and iterative adjacent repetition down at 3.42%.
The mathematical difference comes down to the continuous-time discrete diffusion Evidence Lower Bound (ELBO) derived by Nie et al. (arXiv:2502.09992) and Sahoo et al. (MDLM, arXiv:2406.07524). Instead of fixing t = 0.15, LLaDA samples a continuous mask ratio t ~ Uniform(0, 1] for each training sequence, masks each token independently with probability t, feeds the corrupted sequence noisy_ids into the bidirectional transformer, and scales the cross-entropy loss on masked tokens by 1 / t:
import torchimport torch.nn as nnimport torch.nn.functional as F def compute_llada_diffusion_elbo_loss(model_head, embed, clean_ids, mask_token_id=999, seed=42): """ Continuous-time Masked Diffusion ELBO objective (Nie et al., LLaDA arXiv:2502.09992). 1. Samples t ~ U(0, 1] per sequence and masks tokens in clean_ids -> noisy_ids. 2. Runs forward pass on noisy_ids to obtain logits [B, L, V]. 3. Weights masked token cross-entropy by 1/t. """ gen = torch.Generator(device=clean_ids.device).manual_seed(seed) B, L = clean_ids.shape # 1. Sample continuous mask ratio t ~ U(0, 1] per sequence t = torch.rand((B, 1), generator=gen, device=clean_ids.device).clamp(min=1e-4) # 2. Corrupt each position to [MASK] with probability t BEFORE forward pass rand_u = torch.rand((B, L), generator=gen, device=clean_ids.device) mask_indices = rand_u < t noisy_ids = torch.where(mask_indices, torch.full_like(clean_ids, mask_token_id), clean_ids) # 3. Forward pass on corrupted sequence noisy_ids logits = model_head(embed(noisy_ids)) # 4. Cross-entropy only on masked tokens, weighted by 1/t (exact continuous ELBO) ce_per_token = F.cross_entropy( logits.view(-1, logits.size(-1)), clean_ids.view(-1), reduction="none", ).view(B, L) masked_ce = (ce_per_token * mask_indices.float()) / t loss = masked_ce.sum() / (B * L) return loss, noisy_ids, t # Verify on synthetic batch [B=2, L=64, V=1000]torch.manual_seed(7)dummy_embed = nn.Embedding(1000, 128)dummy_head = nn.Linear(128, 1000)dummy_ids = torch.randint(0, 999, (2, 64))elbo_loss, noisy_sample, sampled_t = compute_llada_diffusion_elbo_loss(dummy_head, dummy_embed, dummy_ids)print(f"Sampled mask ratios t: {[round(v, 3) for v in sampled_t.squeeze().tolist()]}")print(f"Continuous 1/t-weighted ELBO loss: {elbo_loss.item():.4f} nats")
Training across the entire noise schedule t in (0, 1] with the 1 / t variational weight turns a standard bidirectional transformer into a calibrated reverse-diffusion denoiser across every masking ratio from 100% down to 0%.
- Inside the 16-Step Denoising Loop and the 2026 Diffusion LLM Lineup
How does GSAI-ML/LLaDA-8B-Instruct generate a 64-token response at inference time? Starting from a prompt followed by L = 64 [MASK] tokens (t = 1.0), we run S = 16 reverse diffusion steps. At each step, the model predicts x0 = argmax p_theta(x0 | x_t) at all remaining mask positions, scores each prediction by its softmax confidence p_theta(x0 | x_t), locks the top L / S = 4 most confident tokens, and re-masks the remaining uncertain positions for the next step (low-confidence remasking).
Here is the exact token crystallization trace I recorded on our A100 GPU for the prompt “Write a concise Python function binary_search(arr, target) that returns the index of target or -1, followed by a one-sentence complexity note" (warmed steps 2–16 take 40.67–49.56 ms, 42.59 ms on average; step 1 recorded 1,268.95 ms because it includes one-time CUDA kernel warmup):
Step 4 (41.22 ms, 48 masks remaining): Unmasks positions [11, 14, 61, 62] -> [' ', 'ect', '.', '<EOS>']—locking the final sentence period at position 61 while positions 15 through 60 are still [MASK]!
Steps 5–10 (259.61 ms total): Fills in the function body return bisect.bisect_left(arr, target) and the closing code fence, plus the opening words The complexity of this function is O(log n) (note that on a tight 64-token budget, LLaDA-8B takes a shortcut by calling bisect.bisect_left rather than writing out the while loop, which also breaks the prompt's spec, since bisect_left returns an insertion index rather than -1 for a missing target; Section 4 tests the explicit loop).
Steps 11–16 (250.40 ms total): Completes the remaining prose clauses (in the worst case, in the best case, O(1)). The lowest-confidence slots, positions 43–46 at 0.21–0.34 confidence, come out as a glitch: “O(log log the) in the average case”.
A common misconception is that diffusion language models generate words in a uniformly random order across the canvas. When I computed the Spearman rank correlation between canvas position i in {0..63} and crystallization step s in {1..16} (Figure 1, Right), I found a strong positive correlation of rho = 0.7593 (p = 3.56e-13) on the full 64-token canvas, and rho = 0.9259 when using semi-autoregressive 32-token blocks. Because tokens immediately adjacent to the prompt have the strongest local conditioning, low-confidence remasking naturally drifts from left to right—while jumping strictly out of order on 13.49% of all token pairs (272 / 2,016 pairs) to plant high-certainty structural anchors like python, (arr, target, closing periods, and <EOS> first.
The 2025–2026 Open-Weight Diffusion LLM Taxonomy
When evaluating modern diffusion language models on Hugging Face, it is critical to distinguish four distinct architectural families (and separate open-weight checkpoints from closed commercial APIs like Inception Labs’ Mercury 2, Google DeepMind’s Gemini Diffusion, and ByteDance’s Seed-Diffusion Preview):
From-Scratch Dense Bidirectional Diffusion (GSAI-ML/LLaDA-8B-Instruct & GSAI-ML/LLaDA-1.5): 8,015,581,184 parameters (8.016B), 32 layers, hidden_size = 4096, intermediate_size = 12288, 32 attention heads, 32 KV heads, vocab_size = 126464, mask_token_id = 126336. Pre-trained from scratch on 2.3 trillion tokens (arXiv:2502.09992), with LLaDA-1.5 (arXiv:2505.19223) introducing Variance-Reduced Preference Optimization (VRPO) to stabilize RLHF/DPO gradients under random masking.
AR-Initialized Dense Diffusion (Dream-org/Dream-v0-Instruct-7B & apple/DiffuCoder-7B-cpGRPO): 7.62B parameter class (28 layers, hidden_size = 3584, intermediate_size = 18944, 28 query heads, 4 GQA KV heads, vocab_size = 152064, mask_token_id = 151666, matching the 7,615,616,512-parameter Qwen2.5-7B backbone from which they are initialized). Rather than training from random weights, Dream-7B (arXiv:2508.15487) initializes directly from Qwen/Qwen2.5-7B weights and adapts them to bidirectional diffusion via Context-Adaptive Token-Level Noise Rescheduling (CART), retaining the 1-position logit shift of its causal parent.
Block-Diffusion with Hierarchical KV Caching (Efficient-Large-Model/Fast_dLLM_v2_7B & kuleshov-group/bd3lm-owt-block_size16): Building on training-free DualCache (Fast-dLLM v1, arXiv:2505.22618), Fast_dLLM_v2_7B (arXiv:2509.26328) adapts Qwen2.5-7B-Instruct (28 layers, d_model = 3584) into a semi-autoregressive block-diffusion model that maintains an exact causal KV cache across blocks while running parallel bidirectional denoising inside the active block.
Sparse Mixture-of-Experts Diffusion (inclusionAI/LLaDA-MoE-7B-A1B-Instruct): About 7B total parameters with 1.4B active during inference (arXiv:2509.24389, 16 layers, hidden_size = 2048, 64 routed experts with top-8 active routing, vocab_size = 157184). By combining MoE sparsity (56 of 64 routed experts, 87.5%, idle for each token) with parallel unmasking, it slashes active weight reads per denoising step.
- The “San Angeles” Crash: Why 1-Step Unmasking Fails Mathematically
If a bidirectional transformer outputs logits for all 64 masked positions in a single 41-millisecond pass, why can’t we simply take argmax (or sample) at all 64 slots in 1 step (S = 1) and achieve a 64x speedup?
Because a single forward pass of a masked diffusion model parameterizes conditionally independent marginal distributions at each masked position i, rather than their joint distribution:
To prove this quantitatively on the A100, I gave GSAI-ML/LLaDA-8B-Instruct two blank [MASK] [MASK] slots on prompts with two competing two-word answers and inspected the exact step-1 marginal probabilities (Figure 3, Left):
Case 1 — California City (“Los Angeles” vs. “San Francisco”): At Step 1 (S = 1), Slot 1 assigns 70.69% normalized probability to "Los" and 29.31% to "San". Simultaneously, Slot 2 assigns 56.16% probability to "Angeles" and 43.84% to "Francisco". Because Slot 1 and Slot 2 choose independently without seeing each other's realization, the theoretical probability of generating a non-existent chimera ("Los Francisco" or "San Angeles") is 0.7069 * 0.4384 + 0.2931 * 0.5616 = 47.45%. (With greedy argmax this prompt gives "Los Angeles"; the chimera appears when the slots are sampled.) When we use S = 2 steps—locking Slot 1 to "Los" in Pass 1 and running Pass 2 on ["Los", "[MASK]"]—P("Angeles" | "Los") jumps from 56.16% to 99.95% ("Francisco" drops to 0.05%), and P("Francisco" | "San") jumps to 99.37%. The expected hybrid error rate collapses from 47.45% down to 0.22%—roughly a 215x reduction in errors from adding a single conditioning step.
Case 2 — US Metropolis (“New York” vs. “Los Angeles”): At S = 1, Slot 1 predicts "New" (57.55%) vs. "Los" (42.45%), while Slot 2 predicts "York" (70.75%) vs. "Angeles" (29.25%), yielding a 46.87% expected "New Angeles" / "Los York" collision rate. At S = 2, P("York" | "New") reaches 99.99% and P("Angeles" | "Los") reaches 99.86%, dropping the hybrid rate to 0.07%.
Case 3 — AI Research Lab (“Open AI” vs. “Deep Mind”): At S = 1, independent marginals produce "Open Mind" or "Deep AI" 47.23% of the time, which drops to 2.34% at S = 2.
You can reproduce the exact marginal vs. conditional factorization math from our A100 telemetry in PyTorch with this standalone snippet:
import torch # Step-1 marginal and Step-2 conditional probabilities measured on the A100 (dllm_results.json)cases = [ { "name": "California City (Los Angeles vs San Francisco)", "p1": torch.tensor([0.706905, 0.293095]), # Slot 1: P("Los"), P("San") "p2": torch.tensor([0.561611, 0.438389]), # Slot 2: P("Angeles"), P("Francisco") "p2_err_given_s1": torch.tensor([0.000500, 0.006300]), # Step 2: P(wrong | "Los"), P(wrong | "San") }, { "name": "US Metropolis (New York vs Los Angeles)", "p1": torch.tensor([0.575485, 0.424515]), # Slot 1: P("New"), P("Los") "p2": torch.tensor([0.707454, 0.292546]), # Slot 2: P("York"), P("Angeles") "p2_err_given_s1": torch.tensor([0.000100, 0.001400]), # Step 2: P(wrong | "New"), P(wrong | "Los") },] for c in cases: # Off-diagonal outer product gives S=1 independent mode collision rate outer = torch.outer(c["p1"], c["p2"]) s1_chimera = (outer[0, 1] + outer[1, 0]).item() * 100.0 s2_chimera = torch.dot(c["p1"], c["p2_err_given_s1"]).item() * 100.0 print(f"{c['name']}:") print(f" S=1 Independent Chimera Rate : {s1_chimera:.2f}%") print(f" S=2 Conditional Chimera Rate : {s2_chimera:.2f}% ({s1_chimera/s2_chimera:.0f}x error reduction)")
What happens on a full 64-token technical explanation when we use greedy decoding (temperature = 0.0) and sweep the step count S in {1, 2, 4, 8, 16, 32, 64} (Figure 3, Right)? Even without random sampling, unmasking too many tokens per step causes adjacent positions to independently pick the same high-probability words when phrase boundaries can shift by one token (note that single-sample AR-evaluated perplexity is noisy on repetitive strings like S = 2, so adjacent token stutter rate and unique bigram ratio are the primary structural indicators):
1-Step Unmasking (S = 1, 64.0 tokens/step, 49.19 ms): 54.10% adjacent token stutter, 54.10% unique bigrams, generative perplexity 32.03 ("requires a key-value cache because maintain the tokens tokens tokens tokens tokens tokens....... tokens...,,,,, un un un un un un un un parallel parallel").
8-Step Unmasking (S = 8, 8.0 tokens/step, 354.83 ms, 180.37 tok/s): Stutter plunges from 63.93% down to 4.92%, unique bigrams jump to 95.08%, and perplexity drops to 5.86.
16-Step Unmasking (S = 16, 4.0 tokens/step, 711.20 ms, 89.99 tok/s): Adjacent stutter hits 0.0%, mean crystallized confidence reaches 67.62%, and generative perplexity reaches 4.44 (achieving fluent grammatical structure, even though the 8B model’s factual prose mistakenly writes “future tokens” instead of “past tokens”).
32-Step and 64-Step Unmasking (S = 32 in 1,395.26 ms and S = 64 in 2,782.92 ms): 0.0% stutter, unique bigrams 93.44%–98.36%, mean confidence 71.16%–75.73%, and perplexity 6.93 and 5.15.
- Bidirectional Code Infilling and the Fixed-Canvas Span Catch
Because every [MASK] token attends to both preceding and succeeding tokens simultaneously, masked diffusion models perform middle code infilling natively without Fill-In-the-Middle (<fim_prefix>, <fim_suffix>, <fim_middle>) token rearrangements.
However, masked diffusion infilling introduces an engineering constraint that causal FIM models do not have: you must specify the exact number of [MASK] tokens in the gap before running the forward pass.
I tested how GSAI-ML/LLaDA-8B-Instruct handles span-length errors when infilling the missing loop body of binary_search(arr, target) between the prefix (while left <= right:) and the suffix (elif arr[mid] < target: ... return -1), where the ground-truth middle block is exactly 26 tokens (Figure 4, Right):
1.0x Oracle Span (26 [MASK] tokens): LLaDA-8B infills the exact two-line midpoint calculation and target check (mid = (left + right) // 2 followed by if arr[mid] == target: return mid), which compiles cleanly and passes 4 out of 4 functional unit tests.
1.5x Over-Allocated Span (39 [MASK] tokens): What does the model do with 13 extra mask tokens inside a strict Python indentation block? Instead of emitting syntax-breaking garbage, it absorbs the extra 13 tokens by inserting a valid debug print statement — print(f"Element found at index: {mid}")—immediately before return mid. The resulting function still compiles and passes all 4 unit tests.
0.5x Under-Allocated Span (13 [MASK] tokens): Starved of enough tokens to express both the midpoint assignment and the if arr[mid] == target: conditional header, the model writes mid = left + right - left followed by an over-indented return mid, failing Python compilation.
import torch # Verify bidirectional mask canvas construction in PyTorch + all 4 functional unit tests from A100 runprefix_ids = torch.arange(1, 25)suffix_ids = torch.arange(100, 128)for span_len in (13, 26, 39): canvas = torch.cat([prefix_ids, torch.full((span_len,), 126336), suffix_ids]) assert int((canvas == 126336).sum().item()) == span_len prefix = "def binary_search(arr, target):\n left, right = 0, len(arr) - 1\n while left <= right:\n"suffix = " elif arr[mid] < target:\n left = mid + 1\n else:\n right = mid - 1\n return -1" infilled_spans = { "under_0.5x (13 masks)": " mid = left + right - left\n return mid\n", "oracle_1.0x (26 masks)": " mid = (left + right) // 2\n if arr[mid] == target:\n return mid\n", "over_1.5x (39 masks)": ( " mid = (left + right) // 2\n" " if arr[mid] == target:\n" ' print(f"Element found at index: {mid}")\n' " return mid\n" ),} unit_tests = [ ([-1, 0, 3, 5, 9, 12], 9, 4), ([-1, 0, 3, 5, 9, 12], 2, -1), ([-10, -3, 0, 5, 8], -10, 0), ([42], 42, 0),] for label, mid_code in infilled_spans.items(): full_src = prefix + mid_code + suffix try: ns = {} exec(full_src, {}, ns) fn = ns["binary_search"] passed = sum(int(fn(arr, tgt) == exp) for arr, tgt, exp in unit_tests) print(f"{label:24s} -> Compiles: True | Unit Tests Passed: {passed}/{len(unit_tests)}") except SyntaxError as e: print(f"{label:24s} -> Compiles: False | SyntaxError: {e.msg}")
This span-starvation failure is why hybrid edit-diffusion systems like ByteDance’s Seed-Diffusion (arXiv:2508.02193) augment mask-token denoising with Levenshtein insertion and deletion operations during corruption and refinement.
- The Hardware Twist: Why Naive LLaDA Is 2.75x Slower, and How Block KV Caching Fixes It
Here is the benchmark result that surprised me most — and the one almost every surface-level overview of Diffusion LLMs leaves out.
In an autoregressive model like Qwen/Qwen2.5-7B-Instruct, causal masking guarantees that prompt token representations never change during generation. Whether your prompt is 64 tokens or 1,024 tokens, Qwen2.5-7B computes the prompt's keys and values once during prefill and reuses them from the KV cache across all 64 decode steps.
In vanilla GSAI-ML/LLaDA-8B-Instruct, attention is 100% bidirectional (is_causal = False). Although Layer 1's keys and values depend only on the static input embeddings, the moment a single [MASK] token flips to "python" at step 1, every prompt token attends to that newly unmasked word in Layer 1's attention output—which shifts all prompt hidden states in Layers 2 through 32 and invalidates the KV cache across all deeper layers. Consequently, naive LLaDA-8B recomputes the full N_prompt + L sequence from scratch at every denoising step.
Worse, look at the hardware regime transition as N_prompt grows from 64 to 1,024 tokens: at N_prompt = 64..256 (128..320 total tokens), a full bidirectional step takes ~40.5–40.9 ms because it is still partially bound by streaming 16 GB of weights. By N_prompt = 1024 (1,088 total tokens), each bidirectional forward pass executes 2 * 8.016e9 * 1088 = 17.4 TFLOPs and becomes compute-bound on the A100's Tensor Cores, ballooning single-step latency from 40.90 ms to 102.37 ms.
Here are the latencies and throughputs I measured with torch.cuda.Event synchronization across prompt lengths N_prompt in {64, 256, 512, 1024} for generating L = 64 tokens on our A100-SXM4-40GB GPU (Figure 5, Left):
At N_prompt = 64 tokens: Autoregressive Qwen2.5–7B (with KV cache) completes 64 tokens in 2,349.69 ms (27.24 tok/s). Uncached LLaDA-8B takes 2,617.77 ms at S = 64 (24.45 tok/s), 1,315.47 ms at S = 32 (48.65 tok/s), and 653.21 ms at S = 16 (97.98 tok/s). Block-cached LLaDA-8B (b = 16, tau = 0.90, NFE = 10, estimated) finishes in 455.48 ms (140.51 tok/s, or 5.16x faster than AR).
At N_prompt = 256 tokens: Autoregressive Qwen2.5–7B runs at 2,364.87 ms (27.06 tok/s), while uncached LLaDA-8B runs at 2,620.98 ms (24.42 tok/s at S = 64), 1,297.09 ms (49.34 tok/s at S = 32), and 647.82 ms (98.79 tok/s at S = 16). Block-Cached LLaDA-8B finishes in 432.05 ms (148.13 tok/s, or 5.47x faster than AR).
At N_prompt = 512 tokens: Autoregressive Qwen2.5–7B holds steady at 2,364.78 ms (27.06 tok/s). Because uncached LLaDA-8B recomputes all 576 tokens on every pass (59.43 ms/step), it slows to 3,803.77 ms (16.83 tok/s at S = 64), 1,902.22 ms (33.64 tok/s at S = 32), and 951.14 ms (67.29 tok/s at S = 16). Block-Cached LLaDA-8B finishes in 442.31 ms (144.70 tok/s, or 5.35x faster than AR).
At N_prompt = 1024 tokens: Autoregressive Qwen2.5–7B stays flat at 2,381.77 ms (26.87 tok/s). Uncached LLaDA-8B (102.37 ms/step over 1,088 tokens) drops to 6,551.70 ms (9.77 tok/s at S = 64—2.75x slower than AR!), 3,283.68 ms (19.49 tok/s at S = 32—still 1.38x slower than AR), and 1,638.27 ms (39.07 tok/s at S = 16—losing 60% of its short-prompt speed). Block-Cached LLaDA-8B finishes in 495.86 ms (129.07 tok/s—4.80x faster than AR, 3.30x faster than uncached 16-step LLaDA, and 13.21x faster than uncached 64-step LLaDA).
One caveat on the block-cached rows. The LLaDA-8B checkpoint ships without a KV-cache implementation, so I did not time a full cached run end to end. Instead each block-cached total is assembled from measured parts: one timed prompt prefill, plus 10 timed passes over a 16-token block (10 is the pass count a live tau = 0.90 confidence-threshold run actually needed). The block pass was timed without attending to the cached prompt keys, so treat these numbers as a best-case estimate; that is also why 256 tokens comes out slightly faster than 64 (39.16 vs 41.51 ms per block pass is timing noise).
Notice that the speedup comes from two distinct mechanisms working together: on short prompts (N <= 256), confidence-threshold early unmasking (tau = 0.90) provides the primary gain by shrinking S = 64 (or 16) passes down to NFE = 10 passes (6.4 tokens unmasked per pass). On long prompts (N >= 512..1024), Block KV Caching prevents each pass from paying the 102.37 ms full-sequence compute penalty, keeping per-step block latency flat near ~39.8 ms after a single 98.11 ms prefill.
Why is it safe to cache prompt and block keys/values in a diffusion model (Fast-dLLM DualCache / Block Diffusion)? When I hooked all 32 layers of GSAI-ML/LLaDA-8B-Instruct during our 16-step denoising run and computed the cosine similarity of every token's hidden state between step s - 1 and step s (Figure 5, Right), the prompt token representations averaged 98.61% (0.9861) cosine similarity across consecutive steps (97.61% to 99.50% across all transitions), while even the active canvas tokens averaged 94.52% (0.94516) cosine similarity.
import torchimport torch.nn.functional as F @torch.no_grad()def confidence_threshold_block_step(logits_block, block_ids, mask_id=999, tau=0.90): """ Fast-dLLM style confidence-threshold parallel unmasking inside an active block. Unmasks all [MASK] positions with max softmax probability >= tau, guaranteeing at least 1 token (the highest-confidence mask) unmasks per step. """ mask_positions = (block_ids == mask_id) probs = F.softmax(logits_block.float(), dim=-1) top_conf, top_token = probs.max(dim=-1) masked_conf = torch.where(mask_positions, top_conf, torch.tensor(-1.0, device=block_ids.device)) unmask_mask = masked_conf >= tau for b in range(block_ids.size(0)): if mask_positions[b].any() and not unmask_mask[b].any(): best_idx = masked_conf[b].argmax() unmask_mask[b, best_idx] = True updated_block = torch.where(unmask_mask, top_token, block_ids) num_unmasked_now = int(unmask_mask.sum().item()) return updated_block, num_unmasked_now # 1. Execute confidence-threshold block step on a synthetic 16-token blocktorch.manual_seed(42)demo_block = torch.full((1, 16), 999)demo_logits = torch.randn(1, 16, 1000)demo_logits[0, [1, 4, 7, 11, 14], 42] = 12.0 # 5 high-confidence anchor positions (>= 0.90)updated_blk, n_unmasked = confidence_threshold_block_step(demo_logits, demo_block, mask_id=999, tau=0.90)print(f"Active 16-token block unmasked {n_unmasked}/16 tokens in 1 pass (remaining masks: {(updated_blk == 999).sum().item()})") # 2. Replay exact A100 latency accounting at N_prompt = 1024 tokens (L = 64 generated tokens)block_cached_total_ms = 495.86uncached_s16_total_ms = 1638.27uncached_s64_total_ms = 6551.70qwen_ar_total_ms = 2381.77 print(f"N_prompt=1024 | Uncached LLaDA (S=64): {64 / (uncached_s64_total_ms / 1000):.2f} tok/s ({uncached_s64_total_ms:.2f} ms)")print(f"N_prompt=1024 | Qwen2.5-7B AR + Cache : {64 / (qwen_ar_total_ms / 1000):.2f} tok/s ({qwen_ar_total_ms:.2f} ms)")print(f"N_prompt=1024 | Uncached LLaDA (S=16): {64 / (uncached_s16_total_ms / 1000):.2f} tok/s ({uncached_s16_total_ms:.2f} ms)")print(f"N_prompt=1024 | Block-Cached LLaDA : {64 / (block_cached_total_ms / 1000):.2f} tok/s ({block_cached_total_ms:.2f} ms)")
- When to Use Diffusion LLMs vs. Autoregressive LLMs in Production
Our A100 experiments give a concrete decision checklist for choosing between autoregressive decoders and modern block-diffusion language models:
Use Block-Cached Diffusion LLMs (Fast_dLLM_v2_7B, LLaDA-MoE-7B-A1B, Dream-7B) when: You are running low-batch interactive code generation, structured JSON/template completion, or bidirectional middle-of-file code infilling where multiple structural tokens (def, brackets, type annotations, closing delimiters) have high confidence (>0.90) and can unmask 4 to 8 tokens per forward pass—raising single-stream arithmetic intensity on the GPU.
Stick with Causal Autoregressive LLMs (Qwen2.5–7B, Llama-3, Gemma 4) when: You are running high-batch server workloads (batch_size >= 128..200 where autoregressive GEMMs already saturate the A100's ~200 FLOP/byte roofline), variable-length open-ended creative writing where span length is unknown in advance, or long-context workflows without a block-diffusion KV cache implementation.
Never benchmark or deploy a Diffusion LLM with S = 1 step or S = 64 uncached steps: A single step breaks joint token dependencies (“San Angeles”), while naive 64-step full-canvas bidirectional attention recomputes the prompt on every step and falls 2.75x behind causal decoding at 1,024 prompt tokens.
References & Open-Weight Resources
LLaDA (GSAI-ML/LLaDA-8B-Instruct): Shen Nie et al., Large Language Diffusion Models, arXiv:2502.09992 (2025) · Hugging Face Checkpoint
LLaDA 1.5 (GSAI-ML/LLaDA-1.5): Fengqi Zhu et al., LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models, arXiv:2505.19223 (2025) · Hugging Face Checkpoint
LLaDA-MoE (inclusionAI/LLaDA-MoE-7B-A1B-Instruct): Fengqi Zhu et al. (Ant Group & Renmin University), LLaDA-MoE: A Sparse MoE Diffusion Language Model, arXiv:2509.24389 (2025) · Hugging Face Checkpoint
Dream 7B (Dream-org/Dream-v0-Instruct-7B): Jiacheng Ye et al. (HKU NLP & Huawei Noah’s Ark Lab), Dream 7B: Diffusion Large Language Models, arXiv:2508.15487 (2025) · Hugging Face Checkpoint
Fast-dLLM v1 & v2 (Efficient-Large-Model/Fast_dLLM_v2_7B): Chengyue Wu et al. (NVIDIA, MIT & HKU), Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (arXiv:2505.22618) & Fast-dLLM v2: Efficient Block-Diffusion LLM (arXiv:2509.26328, 2025) · Hugging Face Checkpoint
BD3-LMs (kuleshov-group/bd3lm-owt-block_size16): Marianne Arriola et al. (Cornell Tech), Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, arXiv:2503.09573 (ICLR 2025 Oral) · Hugging Face Checkpoint
MDLM: Subham Sekhar Sahoo et al., Simple and Effective Masked Diffusion Language Models, arXiv:2406.07524 (NeurIPS 2024).
DiffuCoder: Shansan Gong et al., DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation, arXiv:2506.20639 (2025).
Seed Diffusion: Yuxuan Song et al., Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference, arXiv:2508.02193 (2025).
Disclosure: All benchmarks, layer-wise cosine similarity traces, multimodality collision rates, and figures in this article were executed and generated by the author on an NVIDIA A100-SXM4–40GB GPU using open-weight Hugging Face checkpoints (GSAI-ML/LLaDA-8B-Instruct, Qwen/Qwen2.5-7B-Instruct, FacebookAI/roberta-large) and PyTorch 2.11.0+cu130. The companion YouTube episode uses AI-synthesized voiceover narration (maya-research/maya1 + SNAC 24kHz) and programmatic Manim animations authored by Arun Prabhakaran (@LatentAILabs).