Speculative decoding: when and why it actually speeds up inference

Your chat endpoint serves 200 requests per second. The model is a 70B Llama 3 fine-tune. The GPU is sitting at 78% utilization, but the user-facing latency is still bad — 380 ms to first token on the median request, 1.1 s P99. The naive read is "we need a bigger box." The actual read is that the GPU is memory-bound, not compute-bound: most of the time is spent shipping weights and KV-cache state from HBM into the SMs, one token at a time, waiting for the next one. Speculative decoding is the technique that turns that one-token-at-a-time pipeline into a several-tokens-at-a-time pipeline without changing what the model actually samples. In our case it dropped p50 TTFT from 380 ms to 140 ms with the same hardware and the same 70B weights.

Here's what it is, what the variants are, and when it stops being a free lunch.

The throughput ceiling for an autoregressive LLM on a single GPU is set by the cost of moving one token's worth of logits and the next token's worth of attention state, not by FLOPs. Doubling the model's parameters roughly doubles the time-per-token on a memory-bound workload, but it does not double the FLOPs the SMs can do — the SMs are sitting idle. Speculative decoding addresses this by doing the heavy forward pass over the target model only every K tokens, and filling the gaps with a much smaller draft model that proposes K tokens in the time the target would have done one.

The property people forget until it bites them: speculative decoding is an exact decoding accelerator. The output distribution is provably identical to running the target model alone, because every proposed token is verified by the target. If the target would have rejected the proposal, the algorithm resamples from a corrected distribution. If the target would have accepted it, the cost of generating it is paid once instead of K times. You don't trade output quality for speed. You trade VRAM and engineering effort for speed.

The original formulation is from DeepMind's Chen, Borgeaud, Irving, Lespiau, and Sifre, "Accelerating Large Language Model Decoding with Speculative Sampling" (Feb 2023). The setup:

The number of accepted tokens per cycle is a random variable. If the draft model is well-aligned with the target — close to it in distribution — the expected accepted length is high and the speedup is high. If they diverge (different tokenizer offset, different training data, different fine-tune), most proposals get rejected and you're paying the draft cost for nothing.

flowchart LR
    A[Prompt] --> B[Draft model Mq<br/>generates K tokens<br/>autoregressively]
    B --> C[Target model Mp<br/>one forward pass<br/>over K+1 positions]
    C --> D{Acceptance<br/>check per token}
    D -- accept --> E[Emit token]
    D -- reject --> F[Resample from<br/>residual distribution]
    E --> G[Loop until EOS]
    F --> G

The cycle cost is roughly: K forward passes of M_q + 1 forward pass of M_p + K cheap logit comparisons. The total time saved per accepted token is the difference between K M_p forward passes (what the unaccelerated decoder would have done) and the actual cycle cost.

This is where the field has moved fast. The naive draft model (e.g. a 1B target for a 70B main) still works, but a few smarter variants have taken over the recommended-default slot. vLLM's speculative decoding docs (v0.22.0, released May 2026) list nine built-in methods; the ones that matter for most teams are these.

The qualitative table in the vLLM docs is sharper than most blog summaries: under low QPS (latency-focused) EAGLE and MTP give the highest gains, while under high QPS (throughput-focused) the gap narrows because the draft cost is amortized. n-gram and suffix give modest, predictable gains across both regimes without a draft model at all.

Here's a real, runnable config that uses EAGLE for offline batched generation. It's straight from the vLLM repo's eagle.md example:

from vllm import LLM, SamplingParams

prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=4,
    speculative_config={
        "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
        "draft_tensor_parallel_size": 1,
        "num_speculative_tokens": 2,
        "method": "eagle",
    },
)

outputs = llm.generate(prompts, sampling_params)

For a server, the CLI form is:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --tensor-parallel-size 4 \
  --speculative-config '{
    "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
    "draft_tensor_parallel_size": 1,
    "num_speculative_tokens": 5,
    "method": "eagle"
  }'

Two notes from running this in production:

num_speculative_tokens

is the K from the algorithm. Default is 5. Setting it too high (8, 16) increases per-cycle cost without proportionally raising acceptance length. Setting it to 2–4 is usually optimal for EAGLE on 7B/8B targets; for 70B targets the optimal K shifts higher.draft_tensor_parallel_size

is the number of GPUs the draft runs on. You do not want the draft to use the same parallelism as the target — that defeats the point. The draft should be on one GPU even when the target spans eight.If you'd rather skip the EAGLE head and just try the n-gram proposer on a code-completion workload:

method: ngram
num_speculative_tokens: 5

No draft model needed, no extra VRAM, no acceptance model. On code with repeated imports and function signatures you'll see a 1.4–1.8x speedup; on open-ended chat you'll see 1.0x and wonder why you bothered.

Speedup is a function of mean accepted tokens per cycle (μ). The relationship for a single-stream workload is roughly:

speedup ≈ (1 + μ) / ( (1 + μ) * draft_cost_ratio + 1 )

where draft_cost_ratio

is the per-token cost of the draft model as a fraction of the target's per-token cost. The graph has a knee around μ = 4 for a draft that costs 10% of the target. If μ falls below 1, the algorithm is a net loss. This is the single number to watch in any benchmark report claiming a "2x speedup from speculative decoding." If they don't report mean accepted tokens, the speedup isn't reproducible.

Measure it. vLLM exposes request-level acceptance rate in examples/features/speculative_decoding/spec_decode_offline.py

. Run it on a representative sample of your traffic before turning the flag on in production. A draft model that scores μ = 4.2 on HumanEval prompts can drop to μ = 1.1 on your support chat corpus. Same weights, different world.

A few traps that bite teams the first time:

num_speculative_tokens

with a weak draft.Speculative decoding is the wrong tool if:

--speculative-config

don't always compose cleanly with the rest of the engine config.Next post: KV cache quantization — how FP8 / INT8 KV caches change the memory budget, and why some of them silently break speculative decoding's acceptance rate.

If you have a draft model recommendation for a target I haven't covered, drop it in the comments — I'm collecting community picks for a follow-up.