{"slug": "speculative-decoding-when-and-why-it-actually-speeds-up-inference", "title": "Speculative decoding: when and why it actually speeds up inference", "summary": "A team running a 70B Llama 3 fine-tune at 200 requests per second cut median time-to-first-token from 380 ms to 140 ms on the same hardware by implementing speculative decoding. The technique addresses memory-bound GPU utilization by running a small draft model to propose multiple tokens per cycle, which the target model then verifies in a single forward pass. The acceleration is provably exact—output distribution remains identical to running the target model alone—but requires the draft model to be closely aligned with the target to avoid wasted computation.", "body_md": "Your chat endpoint serves 200 requests per second. The model is a 70B Llama 3 fine-tune. The GPU is sitting at 78% utilization, but the user-facing latency is still bad — 380 ms to first token on the median request, 1.1 s P99. The naive read is \"we need a bigger box.\" The actual read is that the GPU is *memory-bound*, not compute-bound: most of the time is spent shipping weights and KV-cache state from HBM into the SMs, one token at a time, waiting for the next one. Speculative decoding is the technique that turns that one-token-at-a-time pipeline into a several-tokens-at-a-time pipeline without changing what the model actually samples. In our case it dropped p50 TTFT from 380 ms to 140 ms with the same hardware and the same 70B weights.\n\nHere's what it is, what the variants are, and when it stops being a free lunch.\n\nThe throughput ceiling for an autoregressive LLM on a single GPU is set by the cost of moving one token's worth of logits and the next token's worth of attention state, not by FLOPs. Doubling the model's parameters roughly doubles the time-per-token on a memory-bound workload, but it does *not* double the FLOPs the SMs can do — the SMs are sitting idle. Speculative decoding addresses this by doing the heavy forward pass over the *target* model only every K tokens, and filling the gaps with a much smaller *draft* model that proposes K tokens in the time the target would have done one.\n\nThe property people forget until it bites them: speculative decoding is an **exact** decoding accelerator. The output distribution is provably identical to running the target model alone, because every proposed token is verified by the target. If the target would have rejected the proposal, the algorithm resamples from a corrected distribution. If the target would have accepted it, the cost of generating it is paid once instead of K times. You don't trade output quality for speed. You trade VRAM and engineering effort for speed.\n\nThe original formulation is from DeepMind's Chen, Borgeaud, Irving, Lespiau, and Sifre, [\"Accelerating Large Language Model Decoding with Speculative Sampling\"](https://arxiv.org/abs/2302.01318) (Feb 2023). The setup:\n\nThe number of accepted tokens per cycle is a random variable. If the draft model is well-aligned with the target — close to it in distribution — the expected accepted length is high and the speedup is high. If they diverge (different tokenizer offset, different training data, different fine-tune), most proposals get rejected and you're paying the draft cost for nothing.\n\n``` php\nflowchart LR\n    A[Prompt] --> B[Draft model Mq<br/>generates K tokens<br/>autoregressively]\n    B --> C[Target model Mp<br/>one forward pass<br/>over K+1 positions]\n    C --> D{Acceptance<br/>check per token}\n    D -- accept --> E[Emit token]\n    D -- reject --> F[Resample from<br/>residual distribution]\n    E --> G[Loop until EOS]\n    F --> G\n```\n\nThe cycle cost is roughly: K forward passes of M_q + 1 forward pass of M_p + K cheap logit comparisons. The total time saved per accepted token is the difference between K M_p forward passes (what the unaccelerated decoder would have done) and the actual cycle cost.\n\nThis is where the field has moved fast. The naive draft model (e.g. a 1B target for a 70B main) still works, but a few smarter variants have taken over the recommended-default slot. vLLM's [speculative decoding docs](https://docs.vllm.ai/en/latest/features/spec_decode.html) (v0.22.0, released May 2026) list nine built-in methods; the ones that matter for most teams are these.\n\n| Method | What it is | Best for | Cost / risk |\n|---|---|---|---|\nEAGLE / EAGLE-2 / EAGLE-3 (\n|\nA small head model trained to predict the next layer's hidden state, not the next token. Catches the target model at layer 1 and extrapolates. |\nGeneral-purpose, best raw acceptance length. Recommended default for Llama-style models. | Need a trained EAGLE head per target model. |\nMulti-Token Prediction (MTP) |\nBuilt into the target model itself during training (DeepSeek-V3 style). The model emits several candidate tokens per forward pass. | Targets that ship with native MTP weights. Zero extra parameters. | Not in the open Llama 3 / Mistral / Gemma 2/3 line. |\nN-gram (prompt lookup) |\nNo model. Look up the next N tokens as a suffix in the prompt or recent context. | Code completion, templated outputs, JSON extraction. | Falls off a cliff on free-form prose. |\nSuffix decoding |\nMatch against a suffix tree built from the prompt and recent generations. | Codebases, JSON, anything with repeated structure. | Same as n-gram: useless on chat. |\nMLP speculator |\nA tiny MLP trained on the target's hidden states. | Cases where an EAGLE head is overkill. | Lower acceptance than EAGLE. |\nSelf-speculative / Medusa |\nMultiple prediction heads bolted onto the target. | When you can fine-tune the target. | Adds heads to every forward pass. |\n\nThe qualitative table in the vLLM docs is sharper than most blog summaries: under *low QPS* (latency-focused) EAGLE and MTP give the highest gains, while under *high QPS* (throughput-focused) the gap narrows because the draft cost is amortized. n-gram and suffix give modest, predictable gains across both regimes without a draft model at all.\n\nHere's a real, runnable config that uses EAGLE for offline batched generation. It's straight from the vLLM repo's [eagle.md example](https://github.com/vllm-project/vllm/blob/main/docs/features/speculative_decoding/eagle.md):\n\n``` python\nfrom vllm import LLM, SamplingParams\n\nprompts = [\"The future of AI is\"]\nsampling_params = SamplingParams(temperature=0.8, top_p=0.95)\n\nllm = LLM(\n    model=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n    tensor_parallel_size=4,\n    speculative_config={\n        \"model\": \"yuhuili/EAGLE-LLaMA3-Instruct-8B\",\n        \"draft_tensor_parallel_size\": 1,\n        \"num_speculative_tokens\": 2,\n        \"method\": \"eagle\",\n    },\n)\n\noutputs = llm.generate(prompts, sampling_params)\n```\n\nFor a server, the CLI form is:\n\n```\nvllm serve meta-llama/Meta-Llama-3-8B-Instruct \\\n  --tensor-parallel-size 4 \\\n  --speculative-config '{\n    \"model\": \"yuhuili/EAGLE-LLaMA3-Instruct-8B\",\n    \"draft_tensor_parallel_size\": 1,\n    \"num_speculative_tokens\": 5,\n    \"method\": \"eagle\"\n  }'\n```\n\nTwo notes from running this in production:\n\n`num_speculative_tokens`\n\nis the K from the algorithm. Default is 5. Setting it too high (8, 16) increases per-cycle cost without proportionally raising acceptance length. Setting it to 2–4 is usually optimal for EAGLE on 7B/8B targets; for 70B targets the optimal K shifts higher.`draft_tensor_parallel_size`\n\nis the number of GPUs the draft runs on. You do not want the draft to use the same parallelism as the target — that defeats the point. The draft should be on one GPU even when the target spans eight.If you'd rather skip the EAGLE head and just try the n-gram proposer on a code-completion workload:\n\n```\n# config.yaml — pass with --speculative-config \"$(cat config.yaml)\"\nmethod: ngram\nnum_speculative_tokens: 5\n```\n\nNo draft model needed, no extra VRAM, no acceptance model. On code with repeated imports and function signatures you'll see a 1.4–1.8x speedup; on open-ended chat you'll see 1.0x and wonder why you bothered.\n\nSpeedup is a function of **mean accepted tokens per cycle (μ)**. The relationship for a single-stream workload is roughly:\n\n```\nspeedup ≈ (1 + μ) / ( (1 + μ) * draft_cost_ratio + 1 )\n```\n\nwhere `draft_cost_ratio`\n\nis the per-token cost of the draft model as a fraction of the target's per-token cost. The graph has a knee around μ = 4 for a draft that costs 10% of the target. If μ falls below 1, the algorithm is a net loss. This is the single number to watch in any benchmark report claiming a \"2x speedup from speculative decoding.\" If they don't report mean accepted tokens, the speedup isn't reproducible.\n\nMeasure it. vLLM exposes request-level acceptance rate in `examples/features/speculative_decoding/spec_decode_offline.py`\n\n. Run it on a representative sample of *your* traffic before turning the flag on in production. A draft model that scores μ = 4.2 on HumanEval prompts can drop to μ = 1.1 on your support chat corpus. Same weights, different world.\n\nA few traps that bite teams the first time:\n\n`num_speculative_tokens`\n\nwith a weak draft.Speculative decoding is the wrong tool if:\n\n`--speculative-config`\n\ndon't always compose cleanly with the rest of the engine config.Next post: KV cache quantization — how FP8 / INT8 KV caches change the memory budget, and why some of them silently break speculative decoding's acceptance rate.\n\n*If you have a draft model recommendation for a target I haven't covered, drop it in the comments — I'm collecting community picks for a follow-up.*", "url": "https://wpnews.pro/news/speculative-decoding-when-and-why-it-actually-speeds-up-inference", "canonical_source": "https://dev.to/tech_nuggets/speculative-decoding-when-and-why-it-actually-speeds-up-inference-5pl", "published_at": "2026-06-05 02:15:28+00:00", "updated_at": "2026-06-05 02:41:15.559701+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "ai-infrastructure", "ai-research"], "entities": ["Llama 3", "HBM", "SMs"], "alternates": {"html": "https://wpnews.pro/news/speculative-decoding-when-and-why-it-actually-speeds-up-inference", "markdown": "https://wpnews.pro/news/speculative-decoding-when-and-why-it-actually-speeds-up-inference.md", "text": "https://wpnews.pro/news/speculative-decoding-when-and-why-it-actually-speeds-up-inference.txt", "jsonld": "https://wpnews.pro/news/speculative-decoding-when-and-why-it-actually-speeds-up-inference.jsonld"}}