Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B

A 180-line roofline model built by an independent developer shows speculative decoding on Llama-3-70B collapses at batch 224, turning a 1.92x speedup into a 0.74x slowdown by batch 512, while the same technique on gpt-oss-120b, a mixture-of-experts model with a similar weight footprint, maintains efficiency until batch 1,981. The analysis highlights a 45x difference in crossover points, contradicting published benchmarks from vLLM that report up to 1.5x speedup at low QPS but 1.4x slowdown at high QPS on Llama-3-70B.

Member-only story Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B I built a 180-line roofline model of speculative decoding this week, and the number that fell out of it is batch 224 . That is where speculative decoding stops being a 1.92x speedup on a dense Llama-3–70B and starts being a tax. By batch 512 the model says you are running at 0.74x — you have paid for a draft model, a bigger KV allocation, and a more complicated serving stack in order to serve 26% fewer tokens per second. Then I ran the same model on gpt-oss-120b, a mixture-of-experts model with almost exactly the same weight footprint. The crossover moved to batch 1,981 . Same technique, same hardware class, same draft budget. A 45x difference in where the feature turns on you. The reason this matters is that the published record on speculative decoding openly contradicts itself, and almost every “should I enable this?” blog post cites one half of it. The contradiction nobody reconciles Here is the actual state of the evidence. vLLM’s own blog post, October 2024 , is the canonical warning. On Llama-3–70B with a Qwama-0.5B drafter on 4xH100, they measured up to a 1.5x speedup at QPS=1 — and at high QPS, “we see 1.4x slowdown Llama3–70B on ShareGPT with 4xH100, 1.8x slowdown …