Is Speculative Decoding's Speedup a Hardware Problem or a Model Problem?

A follow-up/sub-part to Part 3 of the LLM inference internals series. Part 3 built sampling-mode speculative decoding with KV caching on both the draft and verifier sides, and it worked correctly, but the speedup it delivered didn't match expectations set by the paper. This post is the record of chasing that gap: every hypothesis tested, which ones were wrong, and what the real answer turned out to be.

The first full gamma sweep, on an open-ended, opinion-style prompt, came back like this:

Slower across the board. The accept/reject math had already been verified correct, with stable, repeatable acceptance rates and no correctness bugs, so this wasn't a bug. It was a real result that needed an explanation.

The first instinct was to blame the hardware. Apple Silicon's MPS backend pays a large, roughly fixed dispatch cost per model call that doesn't shrink proportionally with model size. That was confirmed via isolated microbenchmarks: the 6x smaller draft model was only about 2.8x faster per single-token call (21 to 28ms versus 60 to 90ms). Two sequential model calls per round, each paying that fixed tax, looked like the obvious explanation.

To test this without conflating hardware cost with algorithm cost, I computed a FLOPs-based theoretical speedup using Leviathan et al.'s formula, E[speedup] = (1 - α^(γ+1)) / ((1 - α)(γ·c + 1))

, with a cost ratio c

derived from each model's architecture (layers, hidden size, GQA heads, FLOPs per token via the standard 2 × params

approximation) rather than measured milliseconds. That makes the number hardware-agnostic by construction.

Verifier (Qwen2.5-3B-Instruct):  6,171,394,048 FLOPs/token
Draft   (Qwen2.5-0.5B-Instruct):   987,922,432 FLOPs/token
c = 0.1601

Plugging in the real measured acceptance rates:

This was the first wrong turn. Even with zero hardware overhead, the theoretical ceiling never cleared 1.0x. If MPS overhead were the whole story, the theoretical column should have shown a clear, comfortable win being eaten by dispatch cost. Instead it showed that the algorithm itself, on paper, wasn't going to win at this acceptance rate. Hypothesis 1 was only a partial explanation at best.

If acceptance rate was the real ceiling, the obvious next move was a closer-sized draft: Qwen2.5-1.5B instead of 0.5B, on the theory that a closer size means a closer distribution match to the verifier.

Checking this on paper first, via AutoConfig

with no weights downloaded, gave a worse cost ratio: c = 0.5002

, about 2x cheaper than the verifier instead of 6x. Solving the formula for breakeven acceptance rate at each gamma showed the 1.5B candidate would need 69 to 84 percent acceptance just to reach 1.0x, far above anything measured so far. Second wrong turn, ruled out before spending compute on it. A bigger draft looked like trading one problem, low acceptance, for a worse version of the same problem, an unfavorable cost ratio.

Then I actually ran it. Real numbers, not just theory:

One run cleared 1.0x by a wide margin, the first of the whole investigation. But three others landed back in the 20 to 40 percent range the FLOPs math had already predicted was a losing zone. This wasn't the bigger-draft theory being confirmed. It was acceptance rate itself being far noisier and more content-dependent than a single-number cost-ratio model could capture. The 1.5B draft wasn't reliably better, it was occasionally much better, which pointed at something other than model size driving the variance.

There was also a second anomaly in this data. In several rows, measured speedup exceeded the theoretical ceiling, which is supposedly impossible since theoretical was meant to be a zero-overhead upper bound. That turned out to be a real finding, not a bug. FLOPs-based c

assumes cost scales with parameter count, but on MPS, dispatch overhead compresses the real cost gap between a 1.5B and 3B model far more than FLOPs alone would predict. The FLOPs-based ceiling is a valid algorithmic upper bound, but it is not a valid hardware upper bound on MPS, a subtlety worth naming rather than glossing over.

The high-variance 1.5B result raised the real question directly: what made that one run hit 86 percent acceptance? The recurring guess was entropy. A prompt with structured, predictable, lower-branching continuations should let a small draft model track a large verifier much more closely than an open-ended, creative one.

To test this cleanly, I swept temperature (0.01, 0.3, 0.7) and top_p on the original opinion-style prompt, and acceptance barely moved, staying in the 20 to 45 percent band regardless of sampling settings. Then I swapped the prompt itself: same models, same code, same hyperparameter ranges, but asked for a detailed summary of a structured, factual passage (photosynthesis) instead of an open-ended question, at 500 tokens:

That's the first time changing a single variable moved acceptance meaningfully, and it wasn't temperature or top_p, it was the content itself. Structured, factual text is where this draft and verifier pair actually performs. Open-ended, creative text is where it doesn't. This became the real answer to what causes the acceptance ceiling here: not the models being poorly matched in general, but poorly matched specifically on high-branching-entropy content, a hypothesis Phase 4's entropy correlation study can now test directly instead of guessing at.

With acceptance now reliably in the 45 to 54 percent range and several runs already crossing 1.0x on PyTorch and MPS, the last open question was whether the shortfall from the 1.5 to 2x range reported in the literature was still a hardware story. To check, I ran the same models (GGUF, Q4_K_M quantization) through llama.cpp on the same photosynthesis-style prompt:

n_draft   = 4
n_predict = 504
n_accept  = 391 / 448 drafted  ->  87.277% acceptance
decoded 504 tokens in 10.581s -> 47.6 tok/s (speculative)
verifier-only baseline (same prompt, same runtime): 50.0 tok/s
Speedup: 47.6 / 50.0 ≈ 0.95x

llama.cpp's Metal backend is genuinely faster in absolute terms, roughly 3x the raw tok/s of the PyTorch and MPS setup on both the baseline and the speculative run, confirming the earlier microbenchmark finding that MPS pays real dispatch overhead PyTorch doesn't fully hide. But at 87.3 percent acceptance, well above anything achieved on PyTorch, llama.cpp's own speedup was still around 0.95x, not the 1.5 to 2x hoped for. Fourth wrong turn. If MPS and PyTorch overhead were the primary blocker, a near-zero-overhead runtime at excellent acceptance should have shown a clear win. It didn't.

llama.cpp printed something that pointed at the real remaining factor: 35.7 percent of total wall-clock was unaccounted time, not verifier compute, not draft compute, not sampling, but round-trip and orchestration cost between the draft and verify phases. That's consistent with both implementations, this repo's and llama.cpp's CLI example, running draft and verify strictly sequentially, round after round, rather than pipelining verification of round N with drafting of round N+1.

Putting all four hypotheses together, in order of how much each one turned out to matter:

Leviathan et al.'s headline results didn't come from one trick. They stack several advantages this setup doesn't have:

None of these is a bug in this implementation. They're conditions the paper's results depended on that this project, by design (local, from scratch, consumer hardware, off-the-shelf same-family models), doesn't have.

2 × params

estimate doesn't model attention's quadratic term in sequence length, and the theoretical formula assumes no fixed per-round overhead. Both simplifications that this investigation's own data (measured beating theoretical, llama.cpp's unaccounted time) shows don't fully hold in practice.Series: Part 3, Speculative Decoding: Sampling-Mode Accept/Reject, leads into this post, which leads into Part 4, The Empirical Study (coming)