Autoregressive language model generation is notoriously slow. When you run a 70-billion parameter model on an enterprise GPU, you might get 20 to 30 tokens per second.
The intuitive assumption is that the GPU compute cores are sweating under massive matrix multiplications. But if you profile the GPU during single-batch generation, you find the exact opposite: the GPU's compute cores are sitting 95% idle.
The true bottleneck is memory bandwidth. Speculative decoding is an inference optimization that exploits this hardware reality to double or triple token generation speed with zero loss in mathematical output quality.
Here is what actually happens inside the GPU, the transformer layers, and the sampling math during speculative decoding.
To understand why speculative decoding works, you first have to look at the GPU memory bus.
During autoregressive generation (decoding phase), an LLM generates tokens sequentially: one token per forward pass. To generate a single token for a batch size of 1:
Let us do the concrete math for a 70B parameter model in FP16 running on an Nvidia H100 GPU:
$$\text{Time}_{\text{memory}} = \frac{140\text{ GB}}{3,350\text{ GB/s}} \approx 41.8\text{ ms}$$
Generating 1 token requires roughly $2 \times 70 \times 10^9 = 1.4 \times 10^{11}$ FLOPs.
$$\text{Time}_{\text{compute}} = \frac{1.4 \times 10^{11}\text{ FLOPs}}{1,979 \times 10^{12}\text{ FLOPs/s}} \approx 0.07\text{ ms}$$
The ratio is brutal: the GPU spends 41.8 ms weights from memory and only 0.07 ms doing arithmetic. The arithmetic intensity (FLOPs per byte transferred) is nearly 1, far below the GPU's saturation threshold (which is around 150 FLOPs/byte).
Traditional Autoregressive Decoding (Batch Size = 1):
[ Load 140 GB Weights ] ---> ( Compute 1 Token: 0.07 ms ) ===> Takes 41.8 ms
[ Load 140 GB Weights ] ---> ( Compute 1 Token: 0.07 ms ) ===> Takes 41.8 ms
[ Load 140 GB Weights ] ---> ( Compute 1 Token: 0.07 ms ) ===> Takes 41.8 ms
Result: 3 forward passes = ~125 ms for 3 tokens
What happens if, instead of 1 token, you feed 5 candidate tokens into the 70B target model simultaneously?
The model still loads the 140 GB weights from VRAM exactly once (41.8 ms). But now the Tensor Cores multiply a $[5 \times d]$ matrix against the weights instead of a $[1 \times d]$ vector.
The compute time rises from 0.07 ms to 0.35 ms, but both numbers are dwarfed by the 41.8 ms memory transfer time.
Evaluating 1 Token: 41.8 ms (memory) + 0.07 ms (compute) = 41.87 ms
Evaluating 5 Tokens: 41.8 ms (memory) + 0.35 ms (compute) = 42.15 ms
Verifying 5 tokens in parallel takes virtually the exact same wall-clock time as generating 1 token.
Speculative decoding exploits this asymmetry by splitting generation into two distinct roles:
Here is the exact step-by-step pipeline executed on every speculative decoding iteration:
+-----------------------------------+
| Prompt / Current Prefix Sequence |
+-----------------+-----------------+
|
v
+-----------------------------------+
| Draft Model (e.g. 1B) |
| Generates K tokens sequentially |
| Tokens: [x1, x2, x3, x4, x5] |
+-----------------+-----------------+
|
v
+-----------------------------------+
| Target Model (e.g. 70B) |
| Parallel Forward Pass on prefix |
| + all K speculative tokens |
+-----------------+-----------------+
|
v
+-----------------------------------+
| Speculative Rejection Sampler |
| Accepts x1, x2, x3 |
| Rejects x4 |
| Samples corrected replacement x4'|
| Discards x5 |
+-----------------+-----------------+
|
v
+-----------------------------------+
| Append Accepted Tokens to Prefix |
| Total yield: 4 tokens in 1 step! |
+-------------------+---------------+
A small model (such as a 1B parameter model paired with a 70B target) generates $K$ candidate tokens $(\hat{x}_1, \hat{x}_2, \dots, \hat{x}_K)$ autoregressively.
Because the draft model is tiny (only ~2 GB in FP16), streaming its weights takes under 0.6 ms per token. Generating 5 draft tokens takes roughly $5 \times 0.6\text{ ms} = 3.0\text{ ms}$.
For each generated token, the draft model records its predicted probability distribution $q(x)$.
All $K$ draft tokens are appended to the input sequence and evaluated by the 70B target model in one single forward pass.
Because the attention mask allows each token at position $i$ to attend to all prior tokens, the target model produces the true output probability distributions $p_1(x), p_2(x), \dots, p_K(x)$ for all $K$ positions simultaneously.
We iterate through the draft tokens from $i = 1$ to $K$:
Here is a clean, dependency-free reference implementation of Leviathan & Chen's Speculative Sampling algorithm:
import numpy as np
def speculative_sample_step(draft_probs_list, target_probs_list, draft_tokens):
"""
draft_probs_list: list of K numpy arrays, each shape (vocab_size,) representing q(x)
target_probs_list: list of K+1 numpy arrays, each shape (vocab_size,) representing p(x)
draft_tokens: list of K integers drafted by the small model
Returns: list of accepted and sampled tokens
"""
accepted_tokens = []
K = len(draft_tokens)
for i in range(K):
token = draft_tokens[i]
q_prob = draft_probs_list[i][token]
p_prob = target_probs_list[i][token]
acceptance_threshold = min(1.0, p_prob / (q_prob + 1e-10))
r = np.random.uniform(0.0, 1.0)
if r <= acceptance_threshold:
accepted_tokens.append(token)
else:
residual = np.maximum(0.0, target_probs_list[i] - draft_probs_list[i])
residual_sum = np.sum(residual)
if residual_sum > 0:
p_prime = residual / residual_sum
else:
p_prime = target_probs_list[i]
replacement_token = int(np.random.choice(len(p_prime), p=p_prime))
accepted_tokens.append(replacement_token)
return accepted_tokens
bonus_token = int(np.random.choice(
len(target_probs_list[K]),
p=target_probs_list[K]
))
accepted_tokens.append(bonus_token)
return accepted_tokens
A common misconception among developers is that speculative decoding is an approximation or lossy distillation (like quantization or pruning).
It is mathematically exact. The probability of generating any sequence under speculative decoding is identical to generating directly from the large target model.
Here is the simple algebraic proof:
Let $x$ be the token candidate at position $i$. What is the total probability that token $x$ is emitted?
Probability $x$ was drafted and accepted:
$$P(\text{drafted } x) \times P(\text{accepted} \mid x) = q(x) \times \min\left(1, \frac{p(x)}{q(x)}\right) = \min(p(x), q(x))$$
Probability another token was rejected and $x$ was sampled from $p'(x)$:
The total rejection probability across all possible tokens is:
$$1 - \alpha = 1 - \sum_{y} \min(p(y), q(y)) = \sum_{y} (p(y) - \min(p(y), q(y))) = \sum_{y} \max(0, p(y) - q(y))$$
When rejection happens, the probability of choosing $x$ from the normalized residual is:
$$p'(x) = \frac{\max(0, p(x) - q(x))}{1 - \alpha}$$
Therefore, the joint probability of rejection and choosing $x$ is:
$$(1 - \alpha) \times \frac{\max(0, p(x) - q(x))}{1 - \alpha} = \max(0, p(x) - q(x)) = p(x) - \min(p(x), q(x))$$
The draft distribution $q(x)$ completely cancels out. Whether the draft model has 50% accuracy or 90% accuracy affects only speed, never correctness.
While standalone draft models (like Llama-3-8B drafting for Llama-3-70B) work well, the open-source community and inference engines (vLLM, SGLang, llama.cpp) have developed even faster variations:
Instead of maintaining an entire second model in VRAM, Medusa attaches multiple lightweight feed-forward decoding heads to the target model's final hidden state. Head 1 predicts token $t+1$, Head 2 predicts token $t+2$, Head 3 predicts token $t+3$. This eliminates the draft model footprint entirely.
Standard draft models operate on text tokens, losing the rich contextual embeddings computed by the target model. EAGLE passes the top-layer feature vectors of the target model into the draft head. This boosts token acceptance rates from ~60% to over 80-85%.
Linear drafting guesses a single path $[x_1, x_2, x_3]$. If $x_1$ is rejected, everything else is wasted. Tree attention constructs a tree of multiple speculative branches (e.g. top-2 candidates at each depth). The target model verifies all candidate branches in parallel using a custom 2D attention mask, guaranteeing a higher average accepted token yield per step.
Linear Speculation (Fragile):
[Root] -> x1 -> x2 -> x3 (If x1 fails, x2 and x3 are dead)
Tree Attention Speculation (Resilient):
+--> x1a --> x2a
[Root] --------|
+--> x1b --> x2b
(If x1a fails, x1b might still be accepted!)
Speculative decoding is not a magic bullet for every deployment. Here are the 4 scenarios where it actually hurts performance:
Speculative decoding relies on spare GPU compute. When serving hundreds of concurrent requests (batch size 32, 64, or 128), your GPU is already 100% compute-bound. Adding draft forward passes creates compute contention and reduces overall throughput. Speculative decoding shines primarily at batch sizes 1 to 8 (low-latency streaming).
When sampling with high temperature ($T > 1.0$) on highly creative tasks, the output entropy is high. The overlap $\alpha = \sum \min(p, q)$ shrinks dramatically, causing frequent early rejections. Speculative decoding works best on structured output, code generation, and deterministic reasoning ($T \le 0.7$).
The draft model and target model must share the exact same tokenizer vocabulary and special token mappings. If token IDs do not map to identical string representations, the logits cannot be compared directly.
Running a draft model requires reserving additional GPU VRAM for its weights and its own separate KV cache. On memory-constrained GPUs (e.g. 16 GB or 24 GB consumer cards), this extra memory might push the target model into slower quantized formats or offload buffers.