Speculative Decoding, Explained: The Free Speed Toggle Your Local LLM Is Probably Not Using

There is a setting in LM Studio and llama.cpp that can make a local model generate 1.5 to 2.5 times faster without changing a single word of its output. Not a lower quant, not a smaller model, the same model producing mathematically identical answers, faster. Most people leave it off, and the people who turn it on often pair it wrong and make things slower. It is called speculative decoding, and 2026 is the year it went from a power-user trick to something model makers ship in the box. Here is how it works, what it really delivers, and the cases where it backfires.

The idle-GPU problem it solves #

Start from a fact we keep coming back to on this site: token generation is limited by memory bandwidth, not compute. Each new token forces the GPU to read the model's active weights from VRAM, and as we covered in Bandwidth, Not TFLOPS, the arithmetic units spend most of that time waiting. The same hardware that crawls through generation at 20 tokens per second can chew through a prompt at 800+ tokens per second, because prompt processing evaluates many tokens in one parallel pass while generation does one at a time.

Speculative decoding is the trick that converts generation into something that looks like prompt processing. A small, fast draft model guesses the next several tokens. The big model then checks all of those guesses in a single parallel pass, the kind of work your GPU is good at. Every guess the big model agrees with is accepted instantly. The first guess it disagrees with gets thrown away, and the big model's own choice is used instead.

The part that surprises people: this is lossless. It is not an approximation and there is no quality tradeoff to weigh. The verification step uses a rejection-sampling rule that provably reproduces the big model's output distribution, so you get exactly what the big model alone would have produced. The original Google paper puts it plainly: the speedup comes "without changing the distribution," with "identical outputs." Your only costs are VRAM for the draft model and some extra computation, which is why the technique shines precisely where compute sits idle: single-user local inference.

What the research says it is worth #

Paper-reported figures on datacenter hardware, linked in the sources below. Local speedups are usually smaller; treat 1.5 to 2x as a good outcome on a home box.

The whole game is a single ratio: how often the draft's guesses are accepted, versus what the drafting costs. A draft model that agrees with the big model 70 to 80% of the time roughly doubles your speed. One that agrees half the time barely breaks even, and the drafting overhead can push you underwater. That is why the acceptance rate, which llama.cpp and LM Studio both report, is the first number to check when you try this.

Two practical rules fall out of that ratio. First, the draft must share the big model's vocabulary, which in practice means same family: Qwen drafts for Qwen, Llama drafts for Llama. Second, the draft should be roughly a tenth the size of the main model or smaller. LM Studio's own pairing guidance is a 1B draft for an 8B model, up to about a 1.5B draft for a 32B model. Bigger drafts guess slightly better but cost too much per guess.

The 2026 shift: the draft model now comes in the box #

The reason this technique stopped being niche is called multi-token prediction (MTP). Instead of you hunting for a compatible draft model, the model maker trains a small drafting head into (or alongside) the model itself, so speculation works out of the box with a perfectly matched drafter. DeepSeek-V3's technical report mainstreamed the idea in 2024, training with an MTP objective and noting the extra head "can also be reused for speculative decoding." In 2026 it went fully consumer:

Qwen 3.6 ships MTP layers, and llama.cpp gained support for them this spring (--spec-type mtp

). One owner running the 27B on an M2 Max 96GBreported: "the results are amazing: 2.5x speed increase, bringing it to 28 tok/s". That thread's worked examples (quant choice, KV cache settings, context limits by RAM) are worth reading in full.Gemma 4 shippedofficial MTP drafter checkpointsfor every size in May 2026, down to a 78-million-parameter drafter for the phone-sized E2B. Google claims up to 3x "while guaranteeing the exact same quality as standard generation," and support landed across Ollama, vLLM, SGLang, and MLX.Ollama, which for years had speculative decoding as an open feature request, now supports MTP drafts for Gemma 4 with aDRAFT

command in the Modelfile.

As one r/LocalLLaMA commenter summarized the distinction: "MTP is a type of Speculative Decoding technique. If the traditional spec dec is just to draft perhaps with a standalone drafter, in MTP instead the main model has additional output heads that leverage the main model's state." Same math, better drafter, zero setup.

How to turn it on #

| llama.cpp (classic draft) | `llama-server -m big.gguf --model-draft small.gguf --spec-draft-n-max 3` . Same tokenizer required. |
| llama.cpp (MTP models) | `--spec-type mtp --spec-draft-n-max 3` on an MTP-converted GGUF (Qwen 3.6, Gemma 4). |

| llama.cpp (no draft at all) | --spec-type ngram-mod and friends: drafts by pattern-matching your own recent tokens. Zero extra VRAM, works on any model, helps most when output repeats input (code edits, RAG, summarizing). | | Ollama | Gemma 4 MTP via the DRAFT Modelfile command; classic draft-model support is still limited. |

The n-gram variants deserve a special mention for anyone tight on memory: they cost nothing to try, and for tasks like "edit this function" or "answer from this document," where the model largely retypes text it was given, they can deliver a real speedup with no second model at all. Our runtime guide covers where each of these tools fits.

When it makes things slower #

Speculation is a bet, and the bet does not always pay. Three failure cases show up consistently.

Fast MoE models. This is the big one for 2026 hardware. An independent test this spring ran 19 speculative configurations of Qwen3.6-35B-A3B on a single RTX 3090: baseline 135.7 tokens per second, best speculative result 131.1 (3% slower), worst 15% slower, even with acceptance rates near 100%. The reason is the interaction with Mixture-of-Experts routing: each drafted token can pull a different expert slice into compute, so verifying a batch of drafts loads far more weights than generating one token, and the parallel-verification discount evaporates. The rule of thumb that falls out: the faster your model already is, the less speculation can add. A sparse MoE doing 135 tok/s has little idle bandwidth to reclaim; a dense 70B crawling at 8 tok/s has a lot.

Creative writing. Acceptance rates collapse when there is no single predictable continuation. Deterministic work (code, extraction, structured output, low temperature) drafts beautifully; open-ended prose often does not. LM Studio's documentation says the same: gains vary with how often the draft is right, and deterministic tasks fare best.

VRAM pressure. The draft model and its context live in memory too. If adding a 1B draft forces your main model down a quant tier or shrinks your usable context, you have probably traded away more than you gained. Check the fit math before and after, our calculator makes that quick, and the KV cache guide explains where the hidden memory goes.

The decision cheat-sheet #

The habit that makes this all simple: watch the acceptance rate. Both major runtimes report it. Above roughly 70%, you are winning. Below roughly 60%, the bet is losing and you should change the draft or turn it off. And measure your own tokens per second before and after, on your own workload, because as the 3090 MoE result shows, the only benchmark that counts is the one on your machine.

Sources and how we researched this #

• Founding papers: Leviathan, Kalman & Matias 2022, "Fast Inference from Transformers via Speculative Decoding" (arXiv:2211.17192);Chen et al. 2023, "Accelerating Large Language Model Decoding with Speculative Sampling" (arXiv:2302.01318). - Draft-free lineage: Cai et al. 2024, Medusa (arXiv:2401.10774);Li et al. 2024, EAGLE (arXiv:2401.15077); MTP mainstreamed in theDeepSeek-V3 Technical Report (arXiv:2412.19437). - Primary project docs: llama.cpp speculative-decoding documentation(draft, MTP, EAGLE-3, ngram variants and flags) andLM Studio's speculative decoding guide(pairing rules). - Vendor drafters: Google's Gemma 4 MTP drafter model cards. - Independent and community measurements: the RTX 3090 / Qwen3.6-35B-A3B benchmark(19 configurations, raw JSON published) and the linked r/LocalLLaMA threads. Single-machine results, labeled as such. - We have not benchmarked speculative decoding first-hand. This piece synthesizes the peer-reviewed literature, primary project documentation, and attributed owner reports, cited inline.

*Related: Bandwidth, Not TFLOPS · Prompt processing vs generation · The KV cache, explained · Mixture-of-Experts, explained · *The plain-English quantization guide