Member-only story
I built a 180-line roofline model of speculative decoding this week, and the number that fell out of it is batch 224. That is where speculative decoding stops being a 1.92x speedup on a dense Llama-3–70B and starts being a tax. By batch 512 the model says you are running at 0.74x — you have paid for a draft model, a bigger KV allocation, and a more complicated serving stack in order to serve 26% fewer tokens per second.
Then I ran the same model on gpt-oss-120b, a mixture-of-experts model with almost exactly the same weight footprint. The crossover moved to batch 1,981. Same technique, same hardware class, same draft budget. A 45x difference in where the feature turns on you.
The reason this matters is that the published record on speculative decoding openly contradicts itself, and almost every “should I enable this?” blog post cites one half of it.
The contradiction nobody reconciles #
Here is the actual state of the evidence.
vLLM’s own blog post, October 2024, is the canonical warning. On Llama-3–70B with a Qwama-0.5B drafter on 4xH100, they measured up to a 1.5x speedup at QPS=1 — and at high QPS, “we see 1.4x slowdown Llama3–70B on ShareGPT with 4xH100, 1.8x slowdown…