How Diffusion LLMs Write 64 Words at Once on an A100 GPU

A hands-on benchmark of GSAI-ML/LLaDA-8B-Instruct on an NVIDIA A100-SXM4-40GB GPU found that naive 64-step bidirectional diffusion without a KV cache ran at 9.77 tokens per second on a 1,024-token prompt, 2.75x slower than KV-cached Qwen/Qwen2.5-7B-Instruct at 26.87 tokens per second. Forcing LLaDA-8B to unmask all tokens in a single step produced a 47.45% chance of chimeric word collisions such as "San Angeles" and 54.10% adjacent word stutter, because softmax heads predict independent marginals rather than joint distributions. The author estimates that block-cached prefix reuse plus 0.90 confidence thresholding compresses generation into 10 forward passes, yielding an estimated 129.07 tokens per second on 1,024-token prompts and 148.13 tokens per second at 256 prompt tokens.

Inside LLaDA-8B on a real GPU: out-of-order decoding, the “San Angeles” crash, and why naive bidirectional attention runs 2.75x slower without a KV cache I ran the open-weight diffusion language model GSAI-ML/LLaDA-8B-Instruct against a matched autoregressive twin Qwen/Qwen2.5-7B-Instruct and FacebookAI/roberta-large on an NVIDIA A100-SXM4-40GB GPU, and compared the published configs of Dream-7B, Fast-dLLM v2 and LLaDA-MoE. Unmasking all tokens in a single step crashes into two-word mode collisions "San Angeles": a 47.45% chance when both slots are sampled at once, computed from the model's own probabilities and 54.10% word stutter because softmax heads predict independent marginals rather than joint distributions. Worse, naive 64-step bidirectional diffusion without a KV cache drops to 9.77 tokens per second on a 1,024-token prompt—2.75x slower than eager autoregressive decoding 26.87 tok/s . Block-cached prefix reuse plus 0.90 confidence thresholding compresses generation into 10 forward passes; built from measured prefill and block-pass timings, that gives an estimated 129.07 tokens per second on 1,024-token prompts 148.13 tok/s at 256 prompt tokens . When I first loaded GSAI-ML/LLaDA-8B-Instruct onto an NVIDIA A100-SXM4-40GB GPU and watched it write a 64-token Python binary search function in 16 parallel denoising steps, the very first forward pass broke twenty years of autoregressive intuition. In step 1 one forward pass, about 42 ms once the GPU is warm , the model did not write def at position 0. Instead, out of 64 blank MASK slots mask token id = 126336 , it locked four non-adjacent positions simultaneously: position 1 python , position 6 arr , position 8 target , and position 63 <EOS . At step 2, it filled in position 0 the opening triple-backtick code fence , position 3 def , position 4 binary , and position 7 , . At step 4, it locked the final period . at position 61—anchoring the end of the prose explanation while positions 15 through 60 were still completely blank masks. Then I ran two experiments that exposed the real engineering trade-offs behind the viral “10x faster parallel AI” headlines: First, when I forced LLaDA-8B to unmask all tokens in 1 single step S = 1 , its own probabilities gave a 47.45% chance of splicing competing answers into chimeric hybrids such as "San Angeles" or "Los Francisco" when both slots are sampled at once—and collapsed into 54.10% adjacent word stutter on prose. Second, when I timed vanilla uncached LLaDA-8B S = 64 against its matched autoregressive twin Qwen/Qwen2.5-7B-Instruct on a 1,024-token prompt, naive diffusion was not faster. Because bidirectional attention invalidates the key-value cache on every step and shifts the 1,088-token pass into the compute-bound regime 17.4 TFLOPs per step , uncached 64-step LLaDA-8B ran at 9.77 tokens per second—2.75x slower than KV-cached Qwen2.5-7B at 26.87 tokens per second. Here is everything I measured inside the tensors on our A100 GPU, how modern 2025–2026 diffusion language models actually work under the hood, and how confidence thresholding plus block KV caching turns that long-prompt slowdown into an estimated 4.80x–5.47x speedup over autoregressive decoding. 1. Why Autoregressive Decoding Starves Your GPU — and Why 15% BERT Cannot Generate To see why researchers are replacing the causal decoder mask, start with the hardware arithmetic of Qwen/Qwen2.5-7B-Instruct 7,615,616,512 parameters in bfloat16, occupying 15.23 GB of A100-40GB HBM2 memory at 1,555 GB/s peak bandwidth . During autoregressive generation, causal masking M i, j = -inf for j i ensures that token i only attends to past tokens 1..i. That gives us an exact key-value KV cache, so generating 1 new token only computes queries, keys, values, and feed-forward activations for a single column 1, 1, 3584 . However, to multiply that single vector, the A100 still has to stream all 15.23 GB of model weights out of HBM2 memory on every single token. At batch size 1, arithmetic intensity sits near 1 FLOP per byte far below the A100's ~200 FLOP/byte roofline balance point —leaving the 312 TFLOPS of BF16 Tensor Cores more than 95% idle. Even a fully compiled kernel is hardware-capped near ~100 tokens per second by HBM2 bandwidth 1,555 GB/s / 15.23 GB , and in standard Hugging Face PyTorch eager execution model.generate at batch size 1 , per-layer CUDA kernel launch overhead plus HBM weight reads cap single-stream generation at 26.87–27.24 tokens per second across 64- to 1,024-token prompts 27.06 tok/s at 256 tokens is about 412 GB/s of effective weight bandwidth . If we remove the causal mask is causal = False so the transformer processes 64 token positions in parallel per weight read, why couldn't we use classic masked language models like BERT or RoBERTa-large to generate paragraphs years ago? I tested FacebookAI/roberta-large 355M parameters, trained with a fixed 15% MLM objective against GSAI-ML/LLaDA-8B-Instruct 8.016B parameters, trained with a continuous t ~ U 0, 1 objective across six corruption ratios t from 0.15 15% masked up to 1.00 100% masked across four technical benchmark sentences on the A100 seed 42 . Entropy is measured on the first forward pass for both models; RoBERTa fills all masks in a single pass, while LLaDA fills them over 8 iterative unmasking steps, which is how each model is meant to be used. While the two checkpoints differ in parameter scale, their behavior across t illustrates the structural difference between fixed-ratio MLM pre-training and continuous-time diffusion pre-training: At t = 0.15 1/t = 6.67 : RoBERTa-large recovers 33.33% of masked tokens cleanly with 4.138 bits of predictive entropy and 0.0% adjacent token repetition, while LLaDA-8B recovers 45.83% with 2.413 bits of entropy and 0.0% repetition. At t = 0.30 1/t = 3.33 : RoBERTa-large reaches 44.20% accuracy 4.097 bits entropy, 1.85% repetition , and LLaDA-8B reaches 50.45% accuracy 4.369 bits entropy, 0.0% repetition . At t = 0.50 1/t = 2.00 : RoBERTa-large drops to 13.99% accuracy as entropy climbs to 6.818 bits, while LLaDA-8B achieves 21.19% accuracy at 5.200 bits of entropy. At t = 0.75 1/t = 1.33 : RoBERTa-large entropy surges to 9.271 bits and adjacent word repetition jumps to 23.19%, versus 6.142 bits and 7.58% iterative repetition for LLaDA-8B. At t = 0.90 1/t = 1.11 : Seeing a 90% masked canvas pushes fixed-15% RoBERTa-large far outside its training distribution: its entropy spikes to 11.477 bits, top-1 confidence drops to 5.14%, and 54.73% of its single-pass output collapses into repeated adjacent tokens. At t = 1.00 both models break down in this simple test, with 51.84% and 45.63% repetition: LLaDA gets no chat template here and only 8 steps for the whole sentence, so full generation needs the longer remasking schedule shown in Section 2. Across iterative denoising from t = 0.90, LLaDA-8B holds step-1 entropy at 7.875 bits and iterative adjacent repetition down at 3.42%. The mathematical difference comes down to the continuous-time discrete diffusion Evidence Lower Bound ELBO derived by Nie et al. arXiv:2502.09992 and Sahoo et al. MDLM, arXiv:2406.07524 . Instead of fixing t = 0.15, LLaDA samples a continuous mask ratio t ~ Uniform 0, 1 for each training sequence, masks each token independently with probability t, feeds the corrupted sequence noisy ids into the bidirectional transformer, and scales the cross-entropy loss on masked tokens by 1 / t: python import torchimport torch.nn as nnimport torch.nn.functional as F def compute llada diffusion elbo loss model head, embed, clean ids, mask token id=999, seed=42 : """ Continuous-time Masked Diffusion ELBO objective Nie et al., LLaDA arXiv:2502.09992 . 1. Samples t ~ U 0, 1 per sequence and masks tokens in clean ids - noisy ids. 2. Runs forward pass on noisy ids to obtain logits B, L, V . 3. Weights masked token cross-entropy by 1/t. """ gen = torch.Generator device=clean ids.device .manual seed seed B, L = clean ids.shape 1. Sample continuous mask ratio t ~ U 0, 1 per sequence t = torch.rand B, 1 , generator=gen, device=clean ids.device .clamp min=1e-4 2. Corrupt each position to MASK with probability t BEFORE forward pass rand u = torch.rand B, L , generator=gen, device=clean ids.device mask indices = rand u < t noisy ids = torch.where mask indices, torch.full like clean ids, mask token id , clean ids 3. Forward pass on corrupted sequence noisy ids logits = model head embed noisy ids 4. Cross-entropy only on masked tokens, weighted by 1/t exact continuous ELBO ce per token = F.cross entropy logits.view -1, logits.size -1 , clean ids.view -1 , reduction="none", .view B, L masked ce = ce per token mask indices.float / t loss = masked ce.sum / B L return loss, noisy ids, t Verify on synthetic batch B=2, L=64, V=1000 torch.manual seed 7 dummy embed = nn.Embedding 1000, 128 dummy head = nn.Linear 128, 1000 dummy ids = torch.randint 0, 999, 2, 64 elbo loss, noisy sample, sampled t = compute llada diffusion elbo loss dummy head, dummy embed, dummy ids print f"Sampled mask ratios t: { round v, 3 for v in sampled t.squeeze .tolist }" print f"Continuous 1/t-weighted ELBO loss: {elbo loss.item :.4f} nats" Training across the entire noise schedule t in 0, 1 with the 1 / t variational weight turns a standard bidirectional transformer into a calibrated reverse-diffusion denoiser across every masking ratio from 100% down to 0%. 2. Inside the 16-Step Denoising Loop and the 2026 Diffusion LLM Lineup How does GSAI-ML/LLaDA-8B-Instruct generate a 64-token response at inference time? Starting from a prompt followed by L = 64 MASK tokens t = 1.0 , we run S = 16 reverse diffusion steps. At each step, the model predicts x0 = argmax p theta x0 | x t at all remaining mask positions, scores each prediction by its softmax confidence p theta x0 | x t , locks the top L / S = 4 most confident tokens, and re-masks the remaining uncertain positions for the next step low-confidence remasking . Here is the exact token crystallization trace I recorded on our A100 GPU for the prompt “Write a concise Python function binary search arr, target that returns the index of target or -1, followed by a one-sentence complexity note" warmed steps 2–16 take 40.67–49.56 ms, 42.59 ms on average; step 1 recorded 1,268.95 ms because it includes one-time CUDA kernel warmup : Step 4 41.22 ms, 48 masks remaining : Unmasks positions 11, 14, 61, 62 - ' ', 'ect', '.', '<EOS ' —locking the final sentence period at position 61 while positions 15 through 60 are still MASK Steps 5–10 259.61 ms total : Fills in the function body return bisect.bisect left arr, target and the closing code fence, plus the opening words The complexity of this function is O log n note that on a tight 64-token budget, LLaDA-8B takes a shortcut by calling bisect.bisect left rather than writing out the while loop, which also breaks the prompt's spec, since bisect left returns an insertion index rather than -1 for a missing target; Section 4 tests the explicit loop . Steps 11–16 250.40 ms total : Completes the remaining prose clauses in the worst case, in the best case, O 1 . The lowest-confidence slots, positions 43–46 at 0.21–0.34 confidence, come out as a glitch: “O log log the in the average case”. A common misconception is that diffusion language models generate words in a uniformly random order across the canvas. When I computed the Spearman rank correlation between canvas position i in {0..63} and crystallization step s in {1..16} Figure 1, Right , I found a strong positive correlation of rho = 0.7593 p = 3.56e-13 on the full 64-token canvas, and rho = 0.9259 when using semi-autoregressive 32-token blocks. Because tokens immediately adjacent to the prompt have the strongest local conditioning, low-confidence remasking naturally drifts from left to right—while jumping strictly out of order on 13.49% of all token pairs 272 / 2,016 pairs to plant high-certainty structural anchors like python, arr, target, closing periods, and <EOS first. The 2025–2026 Open-Weight Diffusion LLM Taxonomy When evaluating modern diffusion language models on Hugging Face, it is critical to distinguish four distinct architectural families and separate open-weight checkpoints from closed commercial APIs like Inception Labs’ Mercury 2, Google DeepMind’s Gemini Diffusion, and ByteDance’s Seed-Diffusion Preview : From-Scratch Dense Bidirectional Diffusion GSAI-ML/LLaDA-8B-Instruct & GSAI-ML/LLaDA-1.5 : 8,015,581,184 parameters 8.016B , 32 layers, hidden size = 4096, intermediate size = 12288, 32 attention heads, 32 KV heads, vocab size = 126464, mask token id = 126336. Pre-trained from scratch on 2.3 trillion tokens arXiv:2502.09992 , with LLaDA-1.5 arXiv:2505.19223 introducing Variance-Reduced Preference Optimization VRPO to stabilize RLHF/DPO gradients under random masking. AR-Initialized Dense Diffusion Dream-org/Dream-v0-Instruct-7B & apple/DiffuCoder-7B-cpGRPO : 7.62B parameter class 28 layers, hidden size = 3584, intermediate size = 18944, 28 query heads, 4 GQA KV heads, vocab size = 152064, mask token id = 151666, matching the 7,615,616,512-parameter Qwen2.5-7B backbone from which they are initialized . Rather than training from random weights, Dream-7B arXiv:2508.15487 initializes directly from Qwen/Qwen2.5-7B weights and adapts them to bidirectional diffusion via Context-Adaptive Token-Level Noise Rescheduling CART , retaining the 1-position logit shift of its causal parent. Block-Diffusion with Hierarchical KV Caching Efficient-Large-Model/Fast dLLM v2 7B & kuleshov-group/bd3lm-owt-block size16 : Building on training-free DualCache Fast-dLLM v1, arXiv:2505.22618 , Fast dLLM v2 7B arXiv:2509.26328 adapts Qwen2.5-7B-Instruct 28 layers, d model = 3584 into a semi-autoregressive block-diffusion model that maintains an exact causal KV cache across blocks while running parallel bidirectional denoising inside the active block. Sparse Mixture-of-Experts Diffusion inclusionAI/LLaDA-MoE-7B-A1B-Instruct : About 7B total parameters with 1.4B active during inference arXiv:2509.24389, 16 layers, hidden size = 2048, 64 routed experts with top-8 active routing, vocab size = 157184 . By combining MoE sparsity 56 of 64 routed experts, 87.5%, idle for each token with parallel unmasking, it slashes active weight reads per denoising step. 3. The “San Angeles” Crash: Why 1-Step Unmasking Fails Mathematically If a bidirectional transformer outputs logits for all 64 masked positions in a single 41-millisecond pass, why can’t we simply take argmax or sample at all 64 slots in 1 step S = 1 and achieve a 64x speedup? Because a single forward pass of a masked diffusion model parameterizes conditionally independent marginal distributions at each masked position i, rather than their joint distribution: To prove this quantitatively on the A100, I gave GSAI-ML/LLaDA-8B-Instruct two blank MASK MASK slots on prompts with two competing two-word answers and inspected the exact step-1 marginal probabilities Figure 3, Left : Case 1 — California City “Los Angeles” vs. “San Francisco” : At Step 1 S = 1 , Slot 1 assigns 70.69% normalized probability to "Los" and 29.31% to "San". Simultaneously, Slot 2 assigns 56.16% probability to "Angeles" and 43.84% to "Francisco". Because Slot 1 and Slot 2 choose independently without seeing each other's realization, the theoretical probability of generating a non-existent chimera "Los Francisco" or "San Angeles" is 0.7069 0.4384 + 0.2931 0.5616 = 47.45%. With greedy argmax this prompt gives "Los Angeles"; the chimera appears when the slots are sampled. When we use S = 2 steps—locking Slot 1 to "Los" in Pass 1 and running Pass 2 on "Los", " MASK " —P "Angeles" | "Los" jumps from 56.16% to 99.95% "Francisco" drops to 0.05% , and P "Francisco" | "San" jumps to 99.37%. The expected hybrid error rate collapses from 47.45% down to 0.22%—roughly a 215x reduction in errors from adding a single conditioning step. Case 2 — US Metropolis “New York” vs. “Los Angeles” : At S = 1, Slot 1 predicts "New" 57.55% vs. "Los" 42.45% , while Slot 2 predicts "York" 70.75% vs. "Angeles" 29.25% , yielding a 46.87% expected "New Angeles" / "Los York" collision rate. At S = 2, P "York" | "New" reaches 99.99% and P "Angeles" | "Los" reaches 99.86%, dropping the hybrid rate to 0.07%. Case 3 — AI Research Lab “Open AI” vs. “Deep Mind” : At S = 1, independent marginals produce "Open Mind" or "Deep AI" 47.23% of the time, which drops to 2.34% at S = 2. You can reproduce the exact marginal vs. conditional factorization math from our A100 telemetry in PyTorch with this standalone snippet: python import torch Step-1 marginal and Step-2 conditional probabilities measured on the A100 dllm results.json cases = { "name": "California City Los Angeles vs San Francisco ", "p1": torch.tensor 0.706905, 0.293095 , Slot 1: P "Los" , P "San" "p2": torch.tensor 0.561611, 0.438389 , Slot 2: P "Angeles" , P "Francisco" "p2 err given s1": torch.tensor 0.000500, 0.006300 , Step 2: P wrong | "Los" , P wrong | "San" }, { "name": "US Metropolis New York vs Los Angeles ", "p1": torch.tensor 0.575485, 0.424515 , Slot 1: P "New" , P "Los" "p2": torch.tensor 0.707454, 0.292546 , Slot 2: P "York" , P "Angeles" "p2 err given s1": torch.tensor 0.000100, 0.001400 , Step 2: P wrong | "New" , P wrong | "Los" }, for c in cases: Off-diagonal outer product gives S=1 independent mode collision rate outer = torch.outer c "p1" , c "p2" s1 chimera = outer 0, 1 + outer 1, 0 .item 100.0 s2 chimera = torch.dot c "p1" , c "p2 err given s1" .item 100.0 print f"{c 'name' }:" print f" S=1 Independent Chimera Rate : {s1 chimera:.2f}%" print f" S=2 Conditional Chimera Rate : {s2 chimera:.2f}% {s1 chimera/s2 chimera:.0f}x error reduction " What happens on a full 64-token technical explanation when we use greedy decoding temperature = 0.0 and sweep the step count S in {1, 2, 4, 8, 16, 32, 64} Figure 3, Right ? Even without random sampling, unmasking too many tokens per step causes adjacent positions to independently pick the same high-probability words when phrase boundaries can shift by one token note that single-sample AR-evaluated perplexity is noisy on repetitive strings like S = 2, so adjacent token stutter rate and unique bigram ratio are the primary structural indicators : 1-Step Unmasking S = 1, 64.0 tokens/step, 49.19 ms : 54.10% adjacent token stutter, 54.10% unique bigrams, generative perplexity 32.03 "requires a key-value cache because maintain the tokens tokens tokens tokens tokens tokens....... tokens...,,,,, un un un un un un un un parallel parallel" . 8-Step Unmasking S = 8, 8.0 tokens/step, 354.83 ms, 180.37 tok/s : Stutter plunges from 63.93% down to 4.92%, unique bigrams jump to 95.08%, and perplexity drops to 5.86. 16-Step Unmasking S = 16, 4.0 tokens/step, 711.20 ms, 89.99 tok/s : Adjacent stutter hits 0.0%, mean crystallized confidence reaches 67.62%, and generative perplexity reaches 4.44 achieving fluent grammatical structure, even though the 8B model’s factual prose mistakenly writes “future tokens” instead of “past tokens” . 32-Step and 64-Step Unmasking S = 32 in 1,395.26 ms and S = 64 in 2,782.92 ms : 0.0% stutter, unique bigrams 93.44%–98.36%, mean confidence 71.16%–75.73%, and perplexity 6.93 and 5.15. 4. Bidirectional Code Infilling and the Fixed-Canvas Span Catch Because every MASK token attends to both preceding and succeeding tokens simultaneously, masked diffusion models perform middle code infilling natively without Fill-In-the-Middle <fim prefix , <fim suffix , <fim middle token rearrangements. However, masked diffusion infilling introduces an engineering constraint that causal FIM models do not have: you must specify the exact number of MASK tokens in the gap before running the forward pass. I tested how GSAI-ML/LLaDA-8B-Instruct handles span-length errors when infilling the missing loop body of binary search arr, target between the prefix while left <= right: and the suffix elif arr mid < target: ... return -1 , where the ground-truth middle block is exactly 26 tokens Figure 4, Right : 1.0x Oracle Span 26 MASK tokens : LLaDA-8B infills the exact two-line midpoint calculation and target check mid = left + right // 2 followed by if arr mid == target: return mid , which compiles cleanly and passes 4 out of 4 functional unit tests. 1.5x Over-Allocated Span 39 MASK tokens : What does the model do with 13 extra mask tokens inside a strict Python indentation block? Instead of emitting syntax-breaking garbage, it absorbs the extra 13 tokens by inserting a valid debug print statement — print f"Element found at index: {mid}" —immediately before return mid. The resulting function still compiles and passes all 4 unit tests. 0.5x Under-Allocated Span 13 MASK tokens : Starved of enough tokens to express both the midpoint assignment and the if arr mid == target: conditional header, the model writes mid = left + right - left followed by an over-indented return mid, failing Python compilation. python import torch Verify bidirectional mask canvas construction in PyTorch + all 4 functional unit tests from A100 runprefix ids = torch.arange 1, 25 suffix ids = torch.arange 100, 128 for span len in 13, 26, 39 : canvas = torch.cat prefix ids, torch.full span len, , 126336 , suffix ids assert int canvas == 126336 .sum .item == span len prefix = "def binary search arr, target :\n left, right = 0, len arr - 1\n while left <= right:\n"suffix = " elif arr mid < target:\n left = mid + 1\n else:\n right = mid - 1\n return -1" infilled spans = { "under 0.5x 13 masks ": " mid = left + right - left\n return mid\n", "oracle 1.0x 26 masks ": " mid = left + right // 2\n if arr mid == target:\n return mid\n", "over 1.5x 39 masks ": " mid = left + right // 2\n" " if arr mid == target:\n" ' print f"Element found at index: {mid}" \n' " return mid\n" ,} unit tests = -1, 0, 3, 5, 9, 12 , 9, 4 , -1, 0, 3, 5, 9, 12 , 2, -1 , -10, -3, 0, 5, 8 , -10, 0 , 42 , 42, 0 , for label, mid code in infilled spans.items : full src = prefix + mid code + suffix try: ns = {} exec full src, {}, ns fn = ns "binary search" passed = sum int fn arr, tgt == exp for arr, tgt, exp in unit tests print f"{label:24s} - Compiles: True | Unit Tests Passed: {passed}/{len unit tests }" except SyntaxError as e: print f"{label:24s} - Compiles: False | SyntaxError: {e.msg}" This span-starvation failure is why hybrid edit-diffusion systems like ByteDance’s Seed-Diffusion arXiv:2508.02193 augment mask-token denoising with Levenshtein insertion and deletion operations during corruption and refinement. 5. The Hardware Twist: Why Naive LLaDA Is 2.75x Slower, and How Block KV Caching Fixes It Here is the benchmark result that surprised me most — and the one almost every surface-level overview of Diffusion LLMs leaves out. In an autoregressive model like Qwen/Qwen2.5-7B-Instruct, causal masking guarantees that prompt token representations never change during generation. Whether your prompt is 64 tokens or 1,024 tokens, Qwen2.5-7B computes the prompt's keys and values once during prefill and reuses them from the KV cache across all 64 decode steps. In vanilla GSAI-ML/LLaDA-8B-Instruct, attention is 100% bidirectional is causal = False . Although Layer 1's keys and values depend only on the static input embeddings, the moment a single MASK token flips to "python" at step 1, every prompt token attends to that newly unmasked word in Layer 1's attention output—which shifts all prompt hidden states in Layers 2 through 32 and invalidates the KV cache across all deeper layers. Consequently, naive LLaDA-8B recomputes the full N prompt + L sequence from scratch at every denoising step. Worse, look at the hardware regime transition as N prompt grows from 64 to 1,024 tokens: at N prompt = 64..256 128..320 total tokens , a full bidirectional step takes ~40.5–40.9 ms because it is still partially bound by streaming 16 GB of weights. By N prompt = 1024 1,088 total tokens , each bidirectional forward pass executes 2 8.016e9 1088 = 17.4 TFLOPs and becomes compute-bound on the A100's Tensor Cores, ballooning single-step latency from 40.90 ms to 102.37 ms. Here are the latencies and throughputs I measured with torch.cuda.Event synchronization across prompt lengths N prompt in {64, 256, 512, 1024} for generating L = 64 tokens on our A100-SXM4-40GB GPU Figure 5, Left : At N prompt = 64 tokens: Autoregressive Qwen2.5–7B with KV cache completes 64 tokens in 2,349.69 ms 27.24 tok/s . Uncached LLaDA-8B takes 2,617.77 ms at S = 64 24.45 tok/s , 1,315.47 ms at S = 32 48.65 tok/s , and 653.21 ms at S = 16 97.98 tok/s . Block-cached LLaDA-8B b = 16, tau = 0.90, NFE = 10, estimated finishes in 455.48 ms 140.51 tok/s, or 5.16x faster than AR . At N prompt = 256 tokens: Autoregressive Qwen2.5–7B runs at 2,364.87 ms 27.06 tok/s , while uncached LLaDA-8B runs at 2,620.98 ms 24.42 tok/s at S = 64 , 1,297.09 ms 49.34 tok/s at S = 32 , and 647.82 ms 98.79 tok/s at S = 16 . Block-Cached LLaDA-8B finishes in 432.05 ms 148.13 tok/s, or 5.47x faster than AR . At N prompt = 512 tokens: Autoregressive Qwen2.5–7B holds steady at 2,364.78 ms 27.06 tok/s . Because uncached LLaDA-8B recomputes all 576 tokens on every pass 59.43 ms/step , it slows to 3,803.77 ms 16.83 tok/s at S = 64 , 1,902.22 ms 33.64 tok/s at S = 32 , and 951.14 ms 67.29 tok/s at S = 16 . Block-Cached LLaDA-8B finishes in 442.31 ms 144.70 tok/s, or 5.35x faster than AR . At N prompt = 1024 tokens: Autoregressive Qwen2.5–7B stays flat at 2,381.77 ms 26.87 tok/s . Uncached LLaDA-8B 102.37 ms/step over 1,088 tokens drops to 6,551.70 ms 9.77 tok/s at S = 64—2.75x slower than AR , 3,283.68 ms 19.49 tok/s at S = 32—still 1.38x slower than AR , and 1,638.27 ms 39.07 tok/s at S = 16—losing 60% of its short-prompt speed . Block-Cached LLaDA-8B finishes in 495.86 ms 129.07 tok/s—4.80x faster than AR, 3.30x faster than uncached 16-step LLaDA, and 13.21x faster than uncached 64-step LLaDA . One caveat on the block-cached rows. The LLaDA-8B checkpoint ships without a KV-cache implementation, so I did not time a full cached run end to end. Instead each block-cached total is assembled from measured parts: one timed prompt prefill, plus 10 timed passes over a 16-token block 10 is the pass count a live tau = 0.90 confidence-threshold run actually needed . The block pass was timed without attending to the cached prompt keys, so treat these numbers as a best-case estimate; that is also why 256 tokens comes out slightly faster than 64 39.16 vs 41.51 ms per block pass is timing noise . Notice that the speedup comes from two distinct mechanisms working together: on short prompts N <= 256 , confidence-threshold early unmasking tau = 0.90 provides the primary gain by shrinking S = 64 or 16 passes down to NFE = 10 passes 6.4 tokens unmasked per pass . On long prompts N = 512..1024 , Block KV Caching prevents each pass from paying the 102.37 ms full-sequence compute penalty, keeping per-step block latency flat near ~39.8 ms after a single 98.11 ms prefill. Why is it safe to cache prompt and block keys/values in a diffusion model Fast-dLLM DualCache / Block Diffusion ? When I hooked all 32 layers of GSAI-ML/LLaDA-8B-Instruct during our 16-step denoising run and computed the cosine similarity of every token's hidden state between step s - 1 and step s Figure 5, Right , the prompt token representations averaged 98.61% 0.9861 cosine similarity across consecutive steps 97.61% to 99.50% across all transitions , while even the active canvas tokens averaged 94.52% 0.94516 cosine similarity. python import torchimport torch.nn.functional as F @torch.no grad def confidence threshold block step logits block, block ids, mask id=999, tau=0.90 : """ Fast-dLLM style confidence-threshold parallel unmasking inside an active block. Unmasks all MASK positions with max softmax probability = tau, guaranteeing at least 1 token the highest-confidence mask unmasks per step. """ mask positions = block ids == mask id probs = F.softmax logits block.float , dim=-1 top conf, top token = probs.max dim=-1 masked conf = torch.where mask positions, top conf, torch.tensor -1.0, device=block ids.device unmask mask = masked conf = tau for b in range block ids.size 0 : if mask positions b .any and not unmask mask b .any : best idx = masked conf b .argmax unmask mask b, best idx = True updated block = torch.where unmask mask, top token, block ids num unmasked now = int unmask mask.sum .item return updated block, num unmasked now 1. Execute confidence-threshold block step on a synthetic 16-token blocktorch.manual seed 42 demo block = torch.full 1, 16 , 999 demo logits = torch.randn 1, 16, 1000 demo logits 0, 1, 4, 7, 11, 14 , 42 = 12.0 5 high-confidence anchor positions = 0.90 updated blk, n unmasked = confidence threshold block step demo logits, demo block, mask id=999, tau=0.90 print f"Active 16-token block unmasked {n unmasked}/16 tokens in 1 pass remaining masks: { updated blk == 999 .sum .item } " 2. Replay exact A100 latency accounting at N prompt = 1024 tokens L = 64 generated tokens block cached total ms = 495.86uncached s16 total ms = 1638.27uncached s64 total ms = 6551.70qwen ar total ms = 2381.77 print f"N prompt=1024 | Uncached LLaDA S=64 : {64 / uncached s64 total ms / 1000 :.2f} tok/s {uncached s64 total ms:.2f} ms " print f"N prompt=1024 | Qwen2.5-7B AR + Cache : {64 / qwen ar total ms / 1000 :.2f} tok/s {qwen ar total ms:.2f} ms " print f"N prompt=1024 | Uncached LLaDA S=16 : {64 / uncached s16 total ms / 1000 :.2f} tok/s {uncached s16 total ms:.2f} ms " print f"N prompt=1024 | Block-Cached LLaDA : {64 / block cached total ms / 1000 :.2f} tok/s {block cached total ms:.2f} ms " 6. When to Use Diffusion LLMs vs. Autoregressive LLMs in Production Our A100 experiments give a concrete decision checklist for choosing between autoregressive decoders and modern block-diffusion language models: Use Block-Cached Diffusion LLMs Fast dLLM v2 7B, LLaDA-MoE-7B-A1B, Dream-7B when: You are running low-batch interactive code generation, structured JSON/template completion, or bidirectional middle-of-file code infilling where multiple structural tokens def, brackets, type annotations, closing delimiters have high confidence 0.90 and can unmask 4 to 8 tokens per forward pass—raising single-stream arithmetic intensity on the GPU. Stick with Causal Autoregressive LLMs Qwen2.5–7B, Llama-3, Gemma 4 when: You are running high-batch server workloads batch size = 128..200 where autoregressive GEMMs already saturate the A100's ~200 FLOP/byte roofline , variable-length open-ended creative writing where span length is unknown in advance, or long-context workflows without a block-diffusion KV cache implementation. Never benchmark or deploy a Diffusion LLM with S = 1 step or S = 64 uncached steps: A single step breaks joint token dependencies “San Angeles” , while naive 64-step full-canvas bidirectional attention recomputes the prompt on every step and falls 2.75x behind causal decoding at 1,024 prompt tokens. References & Open-Weight Resources LLaDA GSAI-ML/LLaDA-8B-Instruct : Shen Nie et al., Large Language Diffusion Models, arXiv:2502.09992 2025 · Hugging Face Checkpoint LLaDA 1.5 GSAI-ML/LLaDA-1.5 : Fengqi Zhu et al., LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models, arXiv:2505.19223 2025 · Hugging Face Checkpoint LLaDA-MoE inclusionAI/LLaDA-MoE-7B-A1B-Instruct : Fengqi Zhu et al. Ant Group & Renmin University , LLaDA-MoE: A Sparse MoE Diffusion Language Model, arXiv:2509.24389 2025 · Hugging Face Checkpoint Dream 7B Dream-org/Dream-v0-Instruct-7B : Jiacheng Ye et al. HKU NLP & Huawei Noah’s Ark Lab , Dream 7B: Diffusion Large Language Models, arXiv:2508.15487 2025 · Hugging Face Checkpoint Fast-dLLM v1 & v2 Efficient-Large-Model/Fast dLLM v2 7B : Chengyue Wu et al. NVIDIA, MIT & HKU , Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding arXiv:2505.22618 & Fast-dLLM v2: Efficient Block-Diffusion LLM arXiv:2509.26328, 2025 · Hugging Face Checkpoint BD3-LMs kuleshov-group/bd3lm-owt-block size16 : Marianne Arriola et al. Cornell Tech , Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, arXiv:2503.09573 ICLR 2025 Oral · Hugging Face Checkpoint MDLM: Subham Sekhar Sahoo et al., Simple and Effective Masked Diffusion Language Models, arXiv:2406.07524 NeurIPS 2024 . DiffuCoder: Shansan Gong et al., DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation, arXiv:2506.20639 2025 . Seed Diffusion: Yuxuan Song et al., Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference, arXiv:2508.02193 2025 . Disclosure: All benchmarks, layer-wise cosine similarity traces, multimodality collision rates, and figures in this article were executed and generated by the author on an NVIDIA A100-SXM4–40GB GPU using open-weight Hugging Face checkpoints GSAI-ML/LLaDA-8B-Instruct, Qwen/Qwen2.5-7B-Instruct, FacebookAI/roberta-large and PyTorch 2.11.0+cu130. The companion YouTube episode uses AI-synthesized voiceover narration maya-research/maya1 + SNAC 24kHz and programmatic Manim animations authored by Arun Prabhakaran @LatentAILabs .