Speculative Decoding, Explained: The Free Speed Toggle Your Local LLM Is Probably Not Using

Speculative decoding can speed up local large language model inference by 1.5 to 2.5 times without changing output quality, according to research from Google and DeepMind. The technique uses a small draft model to guess tokens that a larger model verifies in parallel, but requires correct pairing—such as a 1B draft for an 8B model—to avoid slowdowns. LM Studio and llama.cpp support the feature, yet many users leave it off or pair it incorrectly.

There is a setting in LM Studio and llama.cpp that can make a local model generate 1.5 to 2.5 times faster without changing a single word of its output. Not a lower quant, not a smaller model, the same model producing mathematically identical answers, faster. Most people leave it off, and the people who turn it on often pair it wrong and make things slower. It is called speculative decoding, and 2026 is the year it went from a power-user trick to something model makers ship in the box. Here is how it works, what it really delivers, and the cases where it backfires. The idle-GPU problem it solves Start from a fact we keep coming back to on this site: token generation is limited by memory bandwidth, not compute. Each new token forces the GPU to read the model's active weights from VRAM, and as we covered in Bandwidth, Not TFLOPS https://vettedconsumer.com/bandwidth-not-tflops-what-sets-your-local-llm-speed-and-why-the-newest-card-isnt-always-fastest/ , the arithmetic units spend most of that time waiting. The same hardware that crawls through generation at 20 tokens per second can chew through a prompt at 800+ tokens per second, because prompt processing evaluates many tokens in one parallel pass https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/ while generation does one at a time. Speculative decoding is the trick that converts generation into something that looks like prompt processing. A small, fast draft model guesses the next several tokens. The big model then checks all of those guesses in a single parallel pass, the kind of work your GPU is good at. Every guess the big model agrees with is accepted instantly. The first guess it disagrees with gets thrown away, and the big model's own choice is used instead. The part that surprises people: this is lossless . It is not an approximation and there is no quality tradeoff to weigh. The verification step uses a rejection-sampling rule that provably reproduces the big model's output distribution, so you get exactly what the big model alone would have produced. The original Google paper puts it plainly: the speedup comes "without changing the distribution," with "identical outputs." Your only costs are VRAM for the draft model and some extra computation, which is why the technique shines precisely where compute sits idle: single-user local inference. What the research says it is worth | Method paper | Approach | Reported speedup | |---|---|---| | Speculative decoding Leviathan et al., Google, 2022 | Separate small draft model | 2 to 3x on T5X | | Speculative sampling Chen et al., DeepMind, 2023 | Draft model + modified rejection sampling | 2 to 2.5x on Chinchilla 70B | | Medusa Cai et al., 2024 | Extra decoding heads on the model itself, no separate draft | 2.2 to 3.6x | | EAGLE Li et al., 2024 | Tiny drafter that reads the big model's hidden states | 2.7 to 3.5x on Llama 2 70B | Paper-reported figures on datacenter hardware, linked in the sources below. Local speedups are usually smaller; treat 1.5 to 2x as a good outcome on a home box. The whole game is a single ratio: how often the draft's guesses are accepted, versus what the drafting costs. A draft model that agrees with the big model 70 to 80% of the time roughly doubles your speed. One that agrees half the time barely breaks even, and the drafting overhead can push you underwater. That is why the acceptance rate, which llama.cpp and LM Studio both report, is the first number to check when you try this. Two practical rules fall out of that ratio. First, the draft must share the big model's vocabulary, which in practice means same family: Qwen drafts for Qwen, Llama drafts for Llama. Second, the draft should be roughly a tenth the size of the main model or smaller. LM Studio's own pairing guidance is a 1B draft for an 8B model, up to about a 1.5B draft for a 32B model. Bigger drafts guess slightly better but cost too much per guess. The 2026 shift: the draft model now comes in the box The reason this technique stopped being niche is called multi-token prediction MTP . Instead of you hunting for a compatible draft model, the model maker trains a small drafting head into or alongside the model itself, so speculation works out of the box with a perfectly matched drafter. DeepSeek-V3's technical report https://arxiv.org/abs/2412.19437?ref=vettedconsumer.com mainstreamed the idea in 2024, training with an MTP objective and noting the extra head "can also be reused for speculative decoding." In 2026 it went fully consumer: Qwen 3.6 ships MTP layers, and llama.cpp gained support for them this spring --spec-type mtp . One owner running the 27B on an M2 Max 96GB reported https://www.reddit.com/r/LocalLLaMA/comments/1t57xuu/25x faster inference with qwen 36 27b using mtp/?ref=vettedconsumer.com : "the results are amazing: 2.5x speed increase, bringing it to 28 tok/s". That thread's worked examples quant choice, KV cache settings, context limits by RAM are worth reading in full. Gemma 4 shipped official MTP drafter checkpoints https://huggingface.co/google/gemma-4-26B-A4B-it-assistant?ref=vettedconsumer.com for every size in May 2026, down to a 78-million-parameter drafter for the phone-sized E2B. Google claims up to 3x "while guaranteeing the exact same quality as standard generation," and support landed across Ollama, vLLM, SGLang, and MLX. Ollama , which for years had speculative decoding as an open feature request, now supports MTP drafts for Gemma 4 with a DRAFT command in the Modelfile. As one r/LocalLLaMA commenter summarized the distinction https://www.reddit.com/r/LocalLLaMA/comments/1t4jq6h/gemma 4 mtp released/?ref=vettedconsumer.com : "MTP is a type of Speculative Decoding technique. If the traditional spec dec is just to draft perhaps with a standalone drafter, in MTP instead the main model has additional output heads that leverage the main model's state." Same math, better drafter, zero setup. How to turn it on | Runtime | How | |---|---| | LM Studio | Load a main model, pick a same-family draft model in the Speculative Decoding panel e.g. Qwen 2.5 14B + Qwen 2.5 0.5B . It shows accepted-token stats live. | | llama.cpp classic draft | llama-server -m big.gguf --model-draft small.gguf --spec-draft-n-max 3 . Same tokenizer required. | | llama.cpp MTP models | --spec-type mtp --spec-draft-n-max 3 on an MTP-converted GGUF Qwen 3.6, Gemma 4 . | | llama.cpp no draft at all | --spec-type ngram-mod and friends: drafts by pattern-matching your own recent tokens. Zero extra VRAM, works on any model, helps most when output repeats input code edits, RAG, summarizing . | | Ollama | Gemma 4 MTP via the DRAFT Modelfile command; classic draft-model support is still limited. | The n-gram variants deserve a special mention for anyone tight on memory: they cost nothing to try, and for tasks like "edit this function" or "answer from this document," where the model largely retypes text it was given, they can deliver a real speedup with no second model at all. Our runtime guide https://vettedconsumer.com/ollama-vs-lm-studio-vs-llama-cpp-which-local-llm-runtime-should-you-actually-use/ covers where each of these tools fits. When it makes things slower Speculation is a bet, and the bet does not always pay. Three failure cases show up consistently. Fast MoE models. This is the big one for 2026 hardware. An independent test this spring ran 19 speculative configurations of Qwen3.6-35B-A3B on a single RTX 3090 https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090?ref=vettedconsumer.com : baseline 135.7 tokens per second, best speculative result 131.1 3% slower , worst 15% slower, even with acceptance rates near 100%. The reason is the interaction with Mixture-of-Experts routing https://vettedconsumer.com/mixture-of-experts-moe-explained-why-active-parameters-decide-what-runs-on-your-machine/ : each drafted token can pull a different expert slice into compute, so verifying a batch of drafts loads far more weights than generating one token, and the parallel-verification discount evaporates. The rule of thumb that falls out: the faster your model already is, the less speculation can add . A sparse MoE doing 135 tok/s has little idle bandwidth to reclaim; a dense 70B crawling at 8 tok/s has a lot. Creative writing. Acceptance rates collapse when there is no single predictable continuation. Deterministic work code, extraction, structured output, low temperature drafts beautifully; open-ended prose often does not. LM Studio's documentation says the same: gains vary with how often the draft is right, and deterministic tasks fare best. VRAM pressure. The draft model and its context live in memory too. If adding a 1B draft forces your main model down a quant tier or shrinks your usable context, you have probably traded away more than you gained. Check the fit math before and after, our calculator https://vettedconsumer.com/can-i-run-it/ makes that quick, and the KV cache guide https://vettedconsumer.com/the-kv-cache-explained-why-long-context-eats-your-vram-and-how-to-fit-more/ explains where the hidden memory goes. The decision cheat-sheet | Your situation | Verdict | |---|---| | Dense model, feels slow 8 to 30 tok/s , coding or structured tasks | Turn it on. Best case for a 1.5 to 2.5x gain | | Running Qwen 3.6 or Gemma 4 | Use the built-in MTP drafter. Zero pairing work, matched by design | | Code editing / RAG / summarization, any model | Try ngram speculation first. Free, no extra VRAM | | Fast MoE already doing 100+ tok/s | Skip it. Measured results show break-even to slower | | Creative writing at high temperature | Expect little; watch the acceptance rate and turn it off below ~60% | | VRAM already full to the last GB | Skip the draft model; ngram variants only | The habit that makes this all simple: watch the acceptance rate . Both major runtimes report it. Above roughly 70%, you are winning. Below roughly 60%, the bet is losing and you should change the draft or turn it off. And measure your own tokens per second before and after, on your own workload, because as the 3090 MoE result shows, the only benchmark that counts is the one on your machine. Sources and how we researched this - Founding papers: Leviathan, Kalman & Matias 2022, "Fast Inference from Transformers via Speculative Decoding" arXiv:2211.17192 https://arxiv.org/abs/2211.17192?ref=vettedconsumer.com ; Chen et al. 2023, "Accelerating Large Language Model Decoding with Speculative Sampling" arXiv:2302.01318 https://arxiv.org/abs/2302.01318?ref=vettedconsumer.com . - Draft-free lineage: Cai et al. 2024, Medusa arXiv:2401.10774 https://arxiv.org/abs/2401.10774?ref=vettedconsumer.com ; Li et al. 2024, EAGLE arXiv:2401.15077 https://arxiv.org/abs/2401.15077?ref=vettedconsumer.com ; MTP mainstreamed in the DeepSeek-V3 Technical Report arXiv:2412.19437 https://arxiv.org/abs/2412.19437?ref=vettedconsumer.com . - Primary project docs: llama.cpp speculative-decoding documentation https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md?ref=vettedconsumer.com draft, MTP, EAGLE-3, ngram variants and flags and LM Studio's speculative decoding guide https://lmstudio.ai/docs/app/advanced/speculative-decoding?ref=vettedconsumer.com pairing rules . - Vendor drafters: Google's Gemma 4 MTP drafter model cards https://huggingface.co/google/gemma-4-26B-A4B-it-assistant?ref=vettedconsumer.com . - Independent and community measurements: the RTX 3090 / Qwen3.6-35B-A3B benchmark https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090?ref=vettedconsumer.com 19 configurations, raw JSON published and the linked r/LocalLLaMA threads. Single-machine results, labeled as such. - We have not benchmarked speculative decoding first-hand. This piece synthesizes the peer-reviewed literature, primary project documentation, and attributed owner reports, cited inline. Related: Bandwidth, Not TFLOPS · Prompt processing vs generation · The KV cache, explained · Mixture-of-Experts, explained · The plain-English quantization guide