{"slug": "speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss", "title": "Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B", "summary": "A 180-line roofline model built by an independent developer shows speculative decoding on Llama-3-70B collapses at batch 224, turning a 1.92x speedup into a 0.74x slowdown by batch 512, while the same technique on gpt-oss-120b, a mixture-of-experts model with a similar weight footprint, maintains efficiency until batch 1,981. The analysis highlights a 45x difference in crossover points, contradicting published benchmarks from vLLM that report up to 1.5x speedup at low QPS but 1.4x slowdown at high QPS on Llama-3-70B.", "body_md": "Member-only story\n\n# Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B\n\nI built a 180-line roofline model of speculative decoding this week, and the number that fell out of it is **batch 224**. That is where speculative decoding stops being a 1.92x speedup on a dense Llama-3–70B and starts being a tax. By batch 512 the model says you are running at **0.74x** — you have paid for a draft model, a bigger KV allocation, and a more complicated serving stack in order to serve 26% fewer tokens per second.\n\nThen I ran the same model on gpt-oss-120b, a mixture-of-experts model with almost exactly the same weight footprint. The crossover moved to **batch 1,981**. Same technique, same hardware class, same draft budget. A 45x difference in where the feature turns on you.\n\nThe reason this matters is that the published record on speculative decoding openly contradicts itself, and almost every “should I enable this?” blog post cites one half of it.\n\n## The contradiction nobody reconciles\n\nHere is the actual state of the evidence.\n\n**vLLM’s own blog post, October 2024**, is the canonical warning. On Llama-3–70B with a Qwama-0.5B drafter on 4xH100, they measured up to a 1.5x speedup at QPS=1 — and at high QPS, *“we see 1.4x slowdown Llama3–70B on ShareGPT with 4xH100, 1.8x slowdown*…", "url": "https://wpnews.pro/news/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss", "canonical_source": "https://pub.towardsai.net/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss-120b-623812c763de?source=rss----98111c9905da---4", "published_at": "2026-08-01 16:32:19+00:00", "updated_at": "2026-08-01 16:52:27.056322+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["Llama-3-70B", "gpt-oss-120b", "vLLM", "Qwama-0.5B", "H100"], "alternates": {"html": "https://wpnews.pro/news/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss", "markdown": "https://wpnews.pro/news/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss.md", "text": "https://wpnews.pro/news/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss.txt", "jsonld": "https://wpnews.pro/news/speculative-decoding-collapses-at-batch-224-on-llama-3-70b-and-never-on-gpt-oss.jsonld"}}