Speculative decoding: when and why it actually speeds up inference

A team running a 70B Llama 3 fine-tune at 200 requests per second cut median time-to-first-token from 380 ms to 140 ms on the same hardware by implementing speculative decoding. The technique addresses memory-bound GPU utilization by running a small draft model to propose multiple tokens per cycle, which the target model then verifies in a single forward pass. The acceleration is provably exact—output distribution remains identical to running the target model alone—but requires the draft model to be closely aligned with the target to avoid wasted computation.

Your chat endpoint serves 200 requests per second. The model is a 70B Llama 3 fine-tune. The GPU is sitting at 78% utilization, but the user-facing latency is still bad — 380 ms to first token on the median request, 1.1 s P99. The naive read is "we need a bigger box." The actual read is that the GPU is memory-bound , not compute-bound: most of the time is spent shipping weights and KV-cache state from HBM into the SMs, one token at a time, waiting for the next one. Speculative decoding is the technique that turns that one-token-at-a-time pipeline into a several-tokens-at-a-time pipeline without changing what the model actually samples. In our case it dropped p50 TTFT from 380 ms to 140 ms with the same hardware and the same 70B weights. Here's what it is, what the variants are, and when it stops being a free lunch. The throughput ceiling for an autoregressive LLM on a single GPU is set by the cost of moving one token's worth of logits and the next token's worth of attention state, not by FLOPs. Doubling the model's parameters roughly doubles the time-per-token on a memory-bound workload, but it does not double the FLOPs the SMs can do — the SMs are sitting idle. Speculative decoding addresses this by doing the heavy forward pass over the target model only every K tokens, and filling the gaps with a much smaller draft model that proposes K tokens in the time the target would have done one. The property people forget until it bites them: speculative decoding is an exact decoding accelerator. The output distribution is provably identical to running the target model alone, because every proposed token is verified by the target. If the target would have rejected the proposal, the algorithm resamples from a corrected distribution. If the target would have accepted it, the cost of generating it is paid once instead of K times. You don't trade output quality for speed. You trade VRAM and engineering effort for speed. The original formulation is from DeepMind's Chen, Borgeaud, Irving, Lespiau, and Sifre, "Accelerating Large Language Model Decoding with Speculative Sampling" https://arxiv.org/abs/2302.01318 Feb 2023 . The setup: The number of accepted tokens per cycle is a random variable. If the draft model is well-aligned with the target — close to it in distribution — the expected accepted length is high and the speedup is high. If they diverge different tokenizer offset, different training data, different fine-tune , most proposals get rejected and you're paying the draft cost for nothing. php flowchart LR A Prompt -- B Draft model Mq<br/ generates K tokens<br/ autoregressively B -- C Target model Mp<br/ one forward pass<br/ over K+1 positions C -- D{Acceptance<br/ check per token} D -- accept -- E Emit token D -- reject -- F Resample from<br/ residual distribution E -- G Loop until EOS F -- G The cycle cost is roughly: K forward passes of M q + 1 forward pass of M p + K cheap logit comparisons. The total time saved per accepted token is the difference between K M p forward passes what the unaccelerated decoder would have done and the actual cycle cost. This is where the field has moved fast. The naive draft model e.g. a 1B target for a 70B main still works, but a few smarter variants have taken over the recommended-default slot. vLLM's speculative decoding docs https://docs.vllm.ai/en/latest/features/spec decode.html v0.22.0, released May 2026 list nine built-in methods; the ones that matter for most teams are these. | Method | What it is | Best for | Cost / risk | |---|---|---|---| EAGLE / EAGLE-2 / EAGLE-3 | A small head model trained to predict the next layer's hidden state, not the next token. Catches the target model at layer 1 and extrapolates. | General-purpose, best raw acceptance length. Recommended default for Llama-style models. | Need a trained EAGLE head per target model. | Multi-Token Prediction MTP | Built into the target model itself during training DeepSeek-V3 style . The model emits several candidate tokens per forward pass. | Targets that ship with native MTP weights. Zero extra parameters. | Not in the open Llama 3 / Mistral / Gemma 2/3 line. | N-gram prompt lookup | No model. Look up the next N tokens as a suffix in the prompt or recent context. | Code completion, templated outputs, JSON extraction. | Falls off a cliff on free-form prose. | Suffix decoding | Match against a suffix tree built from the prompt and recent generations. | Codebases, JSON, anything with repeated structure. | Same as n-gram: useless on chat. | MLP speculator | A tiny MLP trained on the target's hidden states. | Cases where an EAGLE head is overkill. | Lower acceptance than EAGLE. | Self-speculative / Medusa | Multiple prediction heads bolted onto the target. | When you can fine-tune the target. | Adds heads to every forward pass. | The qualitative table in the vLLM docs is sharper than most blog summaries: under low QPS latency-focused EAGLE and MTP give the highest gains, while under high QPS throughput-focused the gap narrows because the draft cost is amortized. n-gram and suffix give modest, predictable gains across both regimes without a draft model at all. Here's a real, runnable config that uses EAGLE for offline batched generation. It's straight from the vLLM repo's eagle.md example https://github.com/vllm-project/vllm/blob/main/docs/features/speculative decoding/eagle.md : python from vllm import LLM, SamplingParams prompts = "The future of AI is" sampling params = SamplingParams temperature=0.8, top p=0.95 llm = LLM model="meta-llama/Meta-Llama-3-8B-Instruct", tensor parallel size=4, speculative config={ "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B", "draft tensor parallel size": 1, "num speculative tokens": 2, "method": "eagle", }, outputs = llm.generate prompts, sampling params For a server, the CLI form is: vllm serve meta-llama/Meta-Llama-3-8B-Instruct \ --tensor-parallel-size 4 \ --speculative-config '{ "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B", "draft tensor parallel size": 1, "num speculative tokens": 5, "method": "eagle" }' Two notes from running this in production: num speculative tokens is the K from the algorithm. Default is 5. Setting it too high 8, 16 increases per-cycle cost without proportionally raising acceptance length. Setting it to 2–4 is usually optimal for EAGLE on 7B/8B targets; for 70B targets the optimal K shifts higher. draft tensor parallel size is the number of GPUs the draft runs on. You do not want the draft to use the same parallelism as the target — that defeats the point. The draft should be on one GPU even when the target spans eight.If you'd rather skip the EAGLE head and just try the n-gram proposer on a code-completion workload: config.yaml — pass with --speculative-config "$ cat config.yaml " method: ngram num speculative tokens: 5 No draft model needed, no extra VRAM, no acceptance model. On code with repeated imports and function signatures you'll see a 1.4–1.8x speedup; on open-ended chat you'll see 1.0x and wonder why you bothered. Speedup is a function of mean accepted tokens per cycle μ . The relationship for a single-stream workload is roughly: speedup ≈ 1 + μ / 1 + μ draft cost ratio + 1 where draft cost ratio is the per-token cost of the draft model as a fraction of the target's per-token cost. The graph has a knee around μ = 4 for a draft that costs 10% of the target. If μ falls below 1, the algorithm is a net loss. This is the single number to watch in any benchmark report claiming a "2x speedup from speculative decoding." If they don't report mean accepted tokens, the speedup isn't reproducible. Measure it. vLLM exposes request-level acceptance rate in examples/features/speculative decoding/spec decode offline.py . Run it on a representative sample of your traffic before turning the flag on in production. A draft model that scores μ = 4.2 on HumanEval prompts can drop to μ = 1.1 on your support chat corpus. Same weights, different world. A few traps that bite teams the first time: num speculative tokens with a weak draft.Speculative decoding is the wrong tool if: --speculative-config don't always compose cleanly with the rest of the engine config.Next post: KV cache quantization — how FP8 / INT8 KV caches change the memory budget, and why some of them silently break speculative decoding's acceptance rate. If you have a draft model recommendation for a target I haven't covered, drop it in the comments — I'm collecting community picks for a follow-up.