{"slug": "speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm", "title": "Speculative Decoding Explained: How to Get 3× Faster Inference from Any LLM", "summary": "Speculative decoding can deliver 2x to 3x faster large language model inference without retraining or changing output quality, according to an explainer on the technique. The method pairs a small draft model that rapidly proposes four to eight candidate tokens with a large target model that verifies the batch in a single forward pass, preserving the target model's exact output probability distribution. The explainer attributes the speedup to the memory-bandwidth bottleneck of autoregressive generation, where GPU compute units sit idle waiting on weight transfers from memory, and cites Medusa, Eagle and Prompt Lookup Decoding as modern variants.", "body_md": "In this article, you will learn how speculative decoding works, why it produces real speed gains at the hardware level, and what modern implementations look like in practice.\n\nTopics we will cover include:\n\n- The core memory-bandwidth bottleneck that makes standard autoregressive token generation slow.\n- How the draft-and-verify loop works and why it preserves the target model’s output distribution exactly.\n- Modern variants of speculative decoding — including Medusa, Eagle, and Prompt Lookup Decoding — and the practical trade-offs involved in enabling the technique in production.\n\n## Introduction\n\nEvery time you send a message to an AI chatbot, the model generates its response one token at a time. That single constraint is one of the most significant barriers to deploying large language models at scale. As models grow larger and more capable, the time it takes to produce each token becomes a real problem, especially for applications that demand instant feedback like coding assistants, customer service bots, and real-time agents.\n\nSpeculative decoding sidesteps this problem without changing the model, retraining on new data, or sacrificing output quality. It achieves 2x to 3x faster inference purely through a smarter generation strategy. This article breaks down how it works, why it’s faster at a hardware level, and what modern implementations look like in practice.\n\n## The Core Problem: LLMs Generate One Token at a Time\n\nLarge language models generate text autoregressively. The model reads all previous tokens, performs a full computation across its billions of parameters, and produces exactly one new token. Then it does the entire thing again for the next token, and so on, until the response is complete.\n\nThe bottleneck isn’t raw computation speed. Modern AI hardware like NVIDIA H100 GPUs is extraordinarily fast at the kinds of matrix math that power these models. The issue is that the model’s billions of parameters are stored in GPU memory, and before each token can be generated, all of those weights need to be transferred from memory into the processor. This data transfer is the actual constraint, not the computation itself.\n\nGenerating one token at a time means the GPU is repeatedly waiting on memory transfers while its compute units sit largely idle. Engineers call this being memory-bandwidth bound. Adding more raw compute to a server doesn’t meaningfully speed things up, because the limiting factor is how fast data can be moved, not how fast it can be processed.\n\n## What Speculative Decoding Does Differently\n\nThe central idea is straightforward: instead of generating tokens one at a time, generate a batch of candidate tokens all at once and verify them in parallel.\n\nThis works through a two-model system. A small, lightweight model called the draft model rapidly generates a sequence of candidate tokens. These guesses are fast and cheap to produce because the draft model has far fewer parameters than the main model. It still generates tokens autoregressively, just much faster due to its smaller size.\n\nThe large, high-quality model called the target model then evaluates all of those candidate tokens in a single forward pass. Because modern GPUs are built for parallel computation, processing a batch of tokens takes only slightly longer than processing one. The target model checks each candidate and accepts or rejects it based on a probabilistic comparison with its own predictions.\n\nThis process follows a statistical method that provides a mathematical guarantee: the final output has exactly the same probability distribution as if the target model had generated every token on its own. The output quality isn’t approximated. It’s provably identical.\n\n## Walking Through the Speculative Loop\n\nThe draft model generates a short sequence, typically four to eight candidate tokens, very quickly. These represent its best guess at what the target model would say next.\n\nThe target model then takes that entire sequence and scores it in one batch operation. Rather than loading its billions of parameters into the processor once per token, the target model loads them once and evaluates the entire candidate sequence together.\n\nStarting from the first candidate, the target model compares how likely it considers that token versus how likely the draft model considered it. If the target model is at least as confident as the draft model, the token is automatically accepted. If the draft model was more confident, the token is accepted with a probability that decreases as the gap grows. When a token is rejected, a corrected token is sampled from an adjusted distribution that accounts for the difference between the two models’ predictions. This adjustment is precisely what guarantees the final output matches the target model’s distribution exactly, and the loop begins again from there.\n\nIn the best case, all candidate tokens are accepted, plus the target model produces one additional token at the end. This means the system generated five to nine tokens for roughly the cost of one target model pass. In the worst case, the first draft token is rejected and replaced, which actually adds slight overhead compared to standard decoding since the draft model’s generation time still gets paid. Output quality stays intact either way, but practitioners should be aware that the downside of a poorly matched model pair is a small slowdown, not just a neutral result.\n\n## Why This Translates to Real Speed Gains\n\nIn standard generation, loading the target model’s weights from memory dominates the time budget. Each token triggers another expensive memory read, leaving compute units waiting.\n\nWith speculative decoding, the target model’s weights are loaded once per speculative step and then applied across multiple accepted tokens. The GPU’s parallel compute capacity gets to do meaningful work on the entire candidate sequence at once, rather than sitting idle between individual token reads.\n\nThe actual speedup depends primarily on the acceptance rate, meaning how often the draft model’s candidates pass the target model’s verification. When acceptance rates sit around 70 to 80 percent, which is typical for well-matched model pairs on common tasks, the real-world speedup lands between 2x and 3x. Tasks involving predictable or repetitive text tend to produce even higher acceptance rates and larger gains.\n\n## Modern Approaches: Beyond the Two-Model Setup\n\nThe original formulation requires maintaining a separate draft model alongside the target model, which adds memory and engineering overhead. Several newer techniques preserve the core idea while removing the need for a standalone draft model. Worth noting: some of these approaches trade the strict distributional guarantee of the original method for simpler implementation or higher throughput, though the practical impact on output quality is generally negligible.\n\n**[Medusa](https://arxiv.org/abs/2401.10774)** modifies the target model itself by attaching multiple additional prediction heads directly to its final layer. Each head independently predicts a different future token position, so the target model effectively drafts its own candidate sequences without any external model.\n\n**[Eagle](https://arxiv.org/abs/2401.15077)** (Extrapolation Algorithm for Greater Language-model Efficiency) predicts future tokens based on the target model’s internal hidden states rather than its final output tokens. Hidden states carry richer contextual information, which tends to produce higher acceptance rates on complex or domain-specific text.\n\nPrompt Lookup Decoding requires no secondary model at all. It scans the existing prompt and conversation history for phrases likely to recur in the response and uses those directly as draft candidates. This approach works especially well for code editing and document summarization, where the output frequently echoes vocabulary and structure from the input.\n\n## What This Means for Practitioners\n\nSpeculative decoding is available as a built-in option in inference frameworks like **[Hugging Face’s text-generation-inference](https://github.com/huggingface/text-generation-inference)**, **[vLLM](https://github.com/vllm-project/vllm)**, and **[llama.cpp](https://github.com/ggerganov/llama.cpp)**. For most practitioners, enabling it is a configuration change rather than a research project.\n\nThe technique is most beneficial when generating longer outputs, when the model is serving one request at a time rather than large batches, and when the prompt and response share vocabulary or structure. Question answering, document summarization, and code completion tend to see the largest gains. High-throughput batch serving is a different story: in heavily batched settings, speculative decoding can actually hurt throughput because the draft tokens consume memory and compute that could otherwise serve additional requests. Keep that trade-off in mind when choosing whether to enable it in production.\n\n## Conclusion\n\nSpeculative decoding reframes what has long been treated as an unavoidable constraint in language model inference. The one-token-at-a-time process isn’t a fixed requirement but a consequence of how standard generation works, and speculative decoding breaks it by pairing a fast drafting stage with a parallel verification pass that preserves the target model’s output exactly.\n\nThe result is faster inference with no change to model weights, no reduction in output quality, and no modification to the prompting workflow. As the technique becomes more tightly integrated into standard inference tooling, it’s quickly becoming a default part of how large language models are deployed.", "url": "https://wpnews.pro/news/speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm", "canonical_source": "https://machinelearningmastery.com/speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm/", "published_at": "2026-10-23 11:00:00+00:00", "updated_at": "2026-10-23 16:14:08.782344+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "machine-learning"], "entities": ["NVIDIA", "H100", "Medusa", "Eagle", "Prompt Lookup Decoding"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm", "markdown": "https://wpnews.pro/news/speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm.md", "text": "https://wpnews.pro/news/speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm.txt", "jsonld": "https://wpnews.pro/news/speculative-decoding-explained-how-to-get-3x-faster-inference-from-any-llm.jsonld"}}