# Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B

> Source: <https://pub.towardsai.net/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss-120b-623812c763de?source=rss----98111c9905da---4>
> Published: 2026-08-01 16:32:19+00:00

Member-only story

# Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B

I built a 180-line roofline model of speculative decoding this week, and the number that fell out of it is **batch 224**. That is where speculative decoding stops being a 1.92x speedup on a dense Llama-3–70B and starts being a tax. By batch 512 the model says you are running at **0.74x** — you have paid for a draft model, a bigger KV allocation, and a more complicated serving stack in order to serve 26% fewer tokens per second.

Then I ran the same model on gpt-oss-120b, a mixture-of-experts model with almost exactly the same weight footprint. The crossover moved to **batch 1,981**. Same technique, same hardware class, same draft budget. A 45x difference in where the feature turns on you.

The reason this matters is that the published record on speculative decoding openly contradicts itself, and almost every “should I enable this?” blog post cites one half of it.

## The contradiction nobody reconciles

Here is the actual state of the evidence.

**vLLM’s own blog post, October 2024**, is the canonical warning. On Llama-3–70B with a Qwama-0.5B drafter on 4xH100, they measured up to a 1.5x speedup at QPS=1 — and at high QPS, *“we see 1.4x slowdown Llama3–70B on ShareGPT with 4xH100, 1.8x slowdown*…