Speculative Decoding Tutorial: 12 Steps, 75 Min [2026]

A tutorial published on Shattered.io lays out a 12-step, roughly 75-minute harness for measuring speculative decoding speedups across four configurations — a plain baseline, n-gram lookup, a small draft model and EAGLE3 — on the reader's own hardware. The tutorial cites Google's claim of up to a 3x inference-speed improvement from Gemma 4 multi-token-prediction drafters without quality degradation, while noting that figure is a vendor claim and that real gains depend on GPU, prompt mix and concurrency. It gives the acceptance-rate formula (1 – a^(k+1)) / (1 – a), which yields about 3.4 tokens per target pass at a = 0.8 and k = 4 versus about 1.9 at a = 0.5, and warns that speculative decoding helps memory-bound low-QPS workloads but often yields no gain for a saturated multi-tenant API.

Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F How Speculative Decoding Works and When It Pays Off Speculative decoding lets a cheap proposer guess several tokens ahead while the full model checks every guess in a single forward pass, so the text stays the same and the number of expensive passes drops. Google claims up to a 3x inference-speed improvement https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/ from its Gemma 4 multi-token-prediction drafters without degradation in output quality or reasoning logic. That is a vendor claim, and your own number will land somewhere below it or occasionally above it depending on GPU, prompt mix and concurrency. The 12 steps below build a small lab that measures the real speedup on your hardware, in about 75 minutes, with four configurations: a plain baseline, n-gram lookup, a small draft model and EAGLE3. The reason it works is simple. Generating one token with an 8B-parameter model means reading all of its weights from GPU memory, and the arithmetic units sit mostly idle while that happens. Checking five candidate tokens in the same pass costs only slightly more than checking one, because the weights are read once either way. If the proposer guessed right, you bank several tokens for the price of one step. If it guessed wrong, you keep the tokens up to the first mistake and lose a little time on the rest. The open-weight field keeps moving, and the best open source LLM https://shattered.io/best-open-source-llm-october-2026-kimi-glm-deepseek/ options in late 2026 increasingly ship with draft heads or matching drafters. That makes speculative decoding a default thing to evaluate, not an exotic trick. The catch is that every serving stack names the knobs differently, the docs lag the code, and most tutorials online show numbers from a different GPU, a different prompt set and a different library version. A repeatable harness fixes that. The acceptance math you should keep in your head. Call the chance that the target model accepts any single drafted token the acceptance rate, written as a. If the draft proposes k tokens per step and each is accepted independently with probability a, the expected number of tokens produced per target pass is 1 – a^ k+1 / 1 – a . With a = 0.8 and k = 4, that works out to about 3.4 tokens per pass. With a = 0.5 and k = 4, it falls to about 1.9. The drafter is not free, so divide by the cost of the extra work. If one draft step costs 10 percent of a target pass, then k = 4 adds 0.4 of a pass, and the back-of-envelope ceiling at a = 0.8 is about 3.4 divided by 1.4, or about 2.4x. Real systems land below that because verification adds overhead and acceptance is not independent, but the shape holds: acceptance rate dominates everything. When it helps and when it hurts. Decode is memory-bound at small batch sizes, which is where speculation shines. As you pile on concurrent requests, the GPU turns compute-bound, the spare capacity that verification relied on disappears, and rejected tokens become pure waste. The vLLM speculative decoding guide reportedly frames the technique as a way to cut inter-token latency in medium-to-low-QPS, memory-bound workloads, and that matches what the sweep in Step 12 usually shows. A single-user local assistant benefits most. A saturated multi-tenant API often sees no gain at all. The lossless property. In greedy mode temperature 0 , a drafted token is accepted only if it equals the token the target would have picked, so the final text should match a non-speculative run. With sampling, vLLM and similar engines use a rejection-sampling rule that preserves the target distribution. Small numeric differences from batching can still flip a near-tie, so the harness counts identical outputs instead of assuming all of them match. Some stacks also expose opt-in lossy verification modes more on that in the advanced tips , and those break the guarantee on purpose. A worked example before you spend GPU time. The numbers here are hypothetical and exist only to show the reasoning. Suppose a chat reply averages 250 output tokens and one user sees 40 tokens per second without speculation, so a reply takes about 6.3 seconds. A 1.8x per-user gain cuts that to roughly 3.5 seconds, which people notice. Now take a nightly batch job that nobody watches. The same 1.8x gain at a single stream means little there, because the job cares about aggregate throughput at full batch, and that is exactly where speculation tends to fade. The same technique can be a clear win for one product and irrelevant for another, which is why the lab measures both views. Speculative Decoding Methods in vLLM v0.33.0 Compared vLLM’s documentation lists model-based methods EAGLE, MTP and draft models plus n-gram and suffix decoding, all configured through a single --speculative-config JSON argument. The table summarizes how they differ in what you must download and when they tend to win. The last column is analysis based on how each method works, not a measured result, so treat it as a hypothesis for the lab to confirm. | Method | Separate draft checkpoint | Training needed | Likely sweet spot | Config key | |---|---|---|---|---| | Draft model | Yes, a small model with the same tokenizer | No, if a matching small sibling exists | General chat when a same-family small model exists | draft model | | N-gram lookup | No | No | Code edits, JSON extraction, summaries that copy the prompt | ngram | | Suffix decoding | No | No | Agent loops that repeat earlier tool output | suffix | | EAGLE3 | Yes, a head trained for your exact target | Yes, unless a published head exists | Open-ended chat at low concurrency | eagle3 | | MTP | No, uses heads built into the target | Built in by the model maker | Models that ship with native prediction heads | mtp | | Medusa and MLP speculator | Yes | Yes | Legacy setups only | Removed reportedly in 0.33.0 | Draft model. A small sibling of your target proposes tokens, and the big model verifies them. The draft must share the target’s tokenizer, which in practice means the same model family. According to vLLM’s release notes, draft-model speculation reportedly landed on the newer Model Runner V2 in v0.31.0. It is the most intuitive method and the easiest to reason about, but it costs extra VRAM for the draft weights and its own KV cache. N-gram lookup. No model at all. The engine looks at the last few tokens, searches the prompt for the same sequence and proposes whatever followed it there. When the output copies from the input rewriting code, extracting fields from a paragraph, quoting a document acceptance can be very high at almost zero cost. When the output is fresh prose, it proposes nothing useful, and you pay a small lookup tax for nothing. Suffix decoding. A cousin of n-gram lookup that matches against earlier generated text and, in recent vLLM builds, reportedly against a cross-request memory on the GPU. It targets repetitive agent workloads. The lab skips it because its benefit depends on the traffic pattern, but the harness makes it a one-line addition. EAGLE3. A small trained head that reads the target’s hidden states to propose tokens. NVIDIA describes EAGLE3 as a separate draft model that proposes tokens for the larger target to verify, which means you need a published checkpoint that matches your target. The lab uses the vllm-project copy of the EAGLE3 Llama 3.1 8B Instruct checkpoint https://huggingface.co/vllm-project/EAGLE3-LLaMA3.1-Instruct-8B/blob/main/README.md . Its model card says the copy mirrors the yuhuili original at a specific commit, made on 2026-10-30, because vLLM’s CI uses it and the original lives on a personal account. That is a supply-chain decision worth copying in your own pipelines. MTP. Multi-token prediction uses extra heads that the model maker trained into the target itself, so there is no second download. In vLLM it is enabled with the method set to mtp and num speculative tokens set to 1. It only works on models that actually contain those heads. A model without them needs EAGLE or a draft model instead. Medusa and MLP speculators. Older tutorials still show these. The 0.33.0 release notes reportedly removed both, along with the legacy Ray compiled-graph executor. If a guide tells you to launch with a Medusa head, check its date before you spend an evening on it. What changed in the last quarter. Three shifts matter for anyone working from an older guide. First, vLLM v0.33.0 is the current line, built on PyTorch 2.14, and its notes reportedly add refiners for block-diffusion drafters such as DFlash and DSpark, drop Medusa and the MLP speculator and carry an opt-in lossy verification policy. Second, vendors now publish training paths for drafters and not only finished checkpoints. AMD’s ROCm team described EAGLE3 training on Instinct MI350X on October 27, 2026, and NVIDIA’s NeMo AutoModel documentation carries an EAGLE-3 recipe. Third, model makers increasingly ship the drafter alongside the model, as Google did for Gemma 4. The practical effect is that a draft path you ruled out earlier this year may exist today, so re-test instead of relying on memory. Prerequisites and Versions The lab targets a single Linux box with one NVIDIA GPU. Everything below was chosen so that each component has an official source for its current state as of November 13, 2026. | Component | Version used | Notes | |---|---|---| | GPU | NVIDIA, 24 GB of VRAM or more | Weights for an 8B model at 16-bit precision take roughly 16 GB before KV cache. GeForce RTX 5090 is NVIDIA’s current desktop flagship, and any card with enough memory works. | | NVIDIA driver | A current production driver | Install the driver on its own. PyTorch wheels bring their own CUDA runtime, so a full CUDA toolkit is optional for this lab. | | Python | 3.12 | vLLM requires 3.10 or newer and recommends 3.12 or newer. | | vLLM | 0.33.0 | The current release line. It moved to PyTorch 2.14 and dropped CUDA 12.9 support, and it is compatible with CUDA 13.x. | | PyTorch | 2.14 installed by vLLM | Do not pin it yourself. Let the vLLM wheel pick it. | | httpx | Latest | Async HTTP client for the benchmark script. | | Target model | meta-llama/Llama-3.1-8B-Instruct | Gated on Hugging Face. Accept the license first. | | Draft model | meta-llama/Llama-3.2-1B-Instruct | Same tokenizer family as the target. Also gated. | | EAGLE3 head | vllm-project/EAGLE3-LLaMA3.1-Instruct-8B | Trained for the exact Llama 3.1 8B Instruct target. | | uv | Latest | Fast environment manager, optional but used below. | Two honest caveats. First, Llama 3.1 8B Instruct is not a new model, but it is the best-documented target for a three-way comparison because a published EAGLE3 head, a same-tokenizer small sibling and vLLM’s own CI all exist for it. The harness is model-agnostic through three environment variables, so you can swap in a newer target once you have a matching drafter. Second, plan for about 30 GB of free disk for the three checkpoints and the Hugging Face cache, and a Hugging Face token with access to the gated Llama repositories. Record the stack before you run anything. Save the output of nvidia-smi , vllm --version and uv pip freeze next to the results, because a speedup without its stack attached cannot be reproduced and will mislead whoever reads it six months from now. The inference ecosystem moved quickly in the last quarter, with a new vLLM release line, new drafter checkpoints and changed defaults, so a result is only as good as the versions behind it. Steps 1 to 3: Define Metrics, Check the GPU, Build the Environment Step 1: Fix the metrics before you touch the GPU. A benchmark that changes its definition halfway through is useless. The lab records five numbers per run, and the table says what each one answers. | Metric | How the harness computes it | What it tells you | |---|---|---| | Aggregate tok/s | Total completion tokens divided by wall-clock time for the whole batch | Throughput. This is the number a busy server cares about. | | P50 decode tok/s | Per request, tokens minus 1 divided by end minus first token time , then the median | Per-user speed. This is what one person typing into a chat feels. | | P50 and P95 TTFT | Time from request start to first content chunk | Whether speculation hurt the first-token delay. It should not move much. | | Acceptance rate | Accepted draft tokens divided by proposed draft tokens, from the server’s metrics | Why a method won or lost. | | Identical outputs | Count of requests whose text equals the baseline text at temperature 0 | Whether the lossless property held in practice. | Decide now what counts as a win. A reasonable rule: a method earns its place if it improves P50 decode tok/s by a clear margin at the concurrency you actually run, without lowering aggregate throughput at your peak load, and with the identical-output count staying near the full set. Step 2: Check the GPU and driver. Confirm the card is visible, note its memory, and make sure nothing else is holding VRAM. nvidia-smi --query-gpu=name,memory.total,memory.used,driver version --format=csv The output should list your GPU name, total memory, a near-zero used figure and the driver version. If another process holds several gigabytes, stop it now. A draft model and its cache need headroom, and a half-full GPU produces confusing out-of-memory errors in Step 10. Step 3: Create the project and install vLLM 0.33.0. The environment uses uv with Python 3.12. If you prefer pip and venv, the same package names apply. mkdir -p ~/specdec-lab/{data,results,logs} && cd ~/specdec-lab curl -LsSf https://astral.sh/uv/install.sh | sh uv venv --python 3.12 .venv source .venv/bin/activate printf 'vllm==0.33.0\nhttpx\n' requirements.txt uv pip install -r requirements.txt python -c "import vllm, torch; print 'vllm', vllm. version , 'torch', torch. version " Expected output on the last line: vllm 0.33.0 torch 2.14.<patch Your PyTorch patch number may differ. What matters is vllm 0.33.0 and a torch 2.14 build. If the import fails with a CUDA library error, jump to the troubleshooting table, because the usual cause is a driver that is too old for the wheel. Steps 4 and 5: Fetch Checkpoints and Build the Prompt Sets Step 4: Download the three checkpoints. Log in with a read-only token that has access to the Llama repositories, then pull the target, the small draft and the EAGLE3 head. Pre-downloading matters because vLLM would otherwise download during server start, and your health-check loop would time out on a slow connection. export HF TOKEN=hf your read token here hf download meta-llama/Llama-3.1-8B-Instruct hf download meta-llama/Llama-3.2-1B-Instruct hf download vllm-project/EAGLE3-LLaMA3.1-Instruct-8B du -sh ~/.cache/huggingface The Hugging Face command-line tool is called hf in current releases. If your install still exposes the older huggingface-cli name, the subcommand and arguments are the same. The last command prints the cache size so you can see that the three checkpoints landed. Step 5: Generate three prompt workloads with different copy behavior. Speculation lives or dies on how predictable the output is, so one prompt set would hide the story. The generator builds three kinds. Code-edit prompts ask the model to rewrite a short Python function, which means much of the answer repeats the input. Chat prompts ask for fresh explanations, which is the hard case for n-gram lookup. JSON extraction prompts pull fields out of a sentence, which is short and heavily copied. A fixed random seed keeps the set identical across machines. python import json import random from pathlib import Path random.seed 7 OUT = Path "data/prompts.jsonl" OUT.parent.mkdir exist ok=True FUNCS = "def slugify text :\n text = text.strip .lower \n out = \n for ch in text:\n out.append ch if ch.isalnum else '-' \n return ''.join out \n", "def mean values :\n total = 0\n for v in values:\n total += v\n return total / len values \n", "def chunk items, size :\n result = \n for i in range 0, len items , size :\n result.append items i:i + size \n return result\n", "def dedupe items :\n seen = set \n out = \n for x in items:\n if x not in seen:\n seen.add x \n out.append x \n return out\n", "def word count text :\n counts = {}\n for w in text.split :\n counts w = counts.get w, 0 + 1\n return counts\n", "def clamp x, lo, hi :\n if x < lo:\n return lo\n if x hi:\n return hi\n return x\n", TASKS = "Add type hints and a docstring.", "Add input validation that raises ValueError.", "Rewrite it with a comprehension where that is cleaner.", "Add logging calls at the start and the end.", "Add three pytest tests below the function.", "Rename the variables so each name says what it holds.", TOPICS = "database indexes", "TCP congestion control", "CI caching", "feature flags", "rate limiting", "idempotent APIs", "container image layers", "DNS caching", "retry with backoff", "zero-downtime deploys", "observability budgets", "schema migrations", "message queues", "blue-green releases", "secrets rotation", "load balancer health checks", "log sampling", "connection pooling", "canary analysis", "dependency pinning", NAMES = "Ana Ruiz", "Ben Okafor", "Chloe Martin", "Dev Patel", "Elif Demir", "Farid Haddad" CITIES = "Lisbon", "Austin", "Leeds", "Pune", "Gdansk", "Denver", "Osaka", "Tallinn" rows = for i, fn, task in enumerate f, t for f in FUNCS for t in TASKS : prompt = f"Here is a Python function:\n\n{fn}\n{task} Return the full updated code in one block." rows.append {"id": f"code-{i:03d}", "kind": "code", "max tokens": 400, "messages": {"role": "user", "content": prompt} } for i, topic in enumerate TOPICS : for j, tpl in enumerate "Explain {t} to a new engineer in about 200 words.", "Give three pitfalls of {t}, one sentence each, then one fix per pitfall.", : rows.append {"id": f"chat-{i:03d}-{j}", "kind": "chat", "max tokens": 300, "messages": {"role": "user", "content": tpl.format t=topic } } for i in range 36 : text = f"Order {1000 + i 7} was placed by {random.choice NAMES } from " f"{random.choice CITIES } on 2026-{random.randint 1, 11 :02d}-{random.randint 1, 28 :02d} " f"for {random.randint 20, 900 }.{random.randint 0, 99 :02d} USD." prompt = "Extract order id, customer, city, date and amount as one JSON object. " f"Return JSON only.\n\n{text}" rows.append {"id": f"json-{i:03d}", "kind": "json", "max tokens": 120, "messages": {"role": "user", "content": prompt} } OUT.write text "\n".join json.dumps r for r in rows + "\n" print f"wrote {len rows } prompts to {OUT}" for kind in "code", "chat", "json" : print kind, sum 1 for r in rows if r "kind" == kind Run it once. python make prompts.py wrote 112 prompts to data/prompts.jsonl code 36 chat 40 json 36 Synthetic prompts are a starting point, not the finish line. If you serve real traffic, replace data/prompts.jsonl with a sample of it. Public request logs help here, such as the 6.12 billion request LLM dataset from Chutes and Harvard https://shattered.io/chutes-harvard-6-12-billion-llm-dataset-2026/ , which gives you a sense of how varied real prompt and output lengths are. The benchmark client only needs the same three fields per row: an id, a kind, a messages list and a token cap. Steps 6 and 7: Write the Server Launcher and the Streaming Benchmark Client Step 6: One launcher script, four modes. The script maps a mode name to a --speculative-config JSON string, so every run uses identical server flags apart from the speculation setting. It fixes the seed, caps the context at 8192 tokens and leaves 10 percent of VRAM free. The environment variables at the top let you swap the target, the draft or the EAGLE head without editing the file. bash /usr/bin/env bash set -euo pipefail MODE="${1:-baseline}" TARGET="${TARGET:-meta-llama/Llama-3.1-8B-Instruct}" DRAFT="${DRAFT:-meta-llama/Llama-3.2-1B-Instruct}" EAGLE="${EAGLE:-vllm-project/EAGLE3-LLaMA3.1-Instruct-8B}" PORT="${PORT:-8000}" COMMON= --port "$PORT" --max-model-len 8192 --gpu-memory-utilization 0.90 --seed 0 case "$MODE" in baseline EXTRA= ;; ngram EXTRA= --speculative-config '{"method":"ngram","num speculative tokens":5,"prompt lookup max":4}' ;; draft EXTRA= --speculative-config "{\"method\":\"draft model\",\"model\":\"$DRAFT\",\"num speculative tokens\":5}" ;; eagle3 EXTRA= --speculative-config "{\"method\":\"eagle3\",\"model\":\"$EAGLE\",\"num speculative tokens\":3}" ;; echo "unknown mode: $MODE baseline|ngram|draft|eagle3 " &2; exit 2 ;; esac exec vllm serve "$TARGET" "${COMMON @ }" ${EXTRA @ +"${EXTRA @ }"} Three details are worth explaining. The JSON goes inside single quotes for the static modes so the shell leaves the double quotes alone, and inside an escaped double-quoted string where a variable must expand. The ${EXTRA @ +"${EXTRA @ }"} expansion keeps the baseline mode working on older bash versions, which treat an empty array as unset under set -u . And exec replaces the shell with the server process, so the process ID you capture later is the server’s own. The num speculative tokens values 5 for n-gram and draft, 3 for EAGLE3 are starting points, not truths. Step 12 and the advanced tips show how to sweep them. Step 7: A streaming client that measures what users feel. The benchmark sends each prompt to the OpenAI-compatible chat endpoint with temperature 0 and streaming on. It records time to first token, total time, text and the token count. One design choice matters a lot: the client reads the token count from the usage block that the server sends at the end of the stream, because a single streamed chunk can carry several tokens when speculation accepts a run of them. Counting chunks would make speculative runs look slower than they are. python import argparse import asyncio import json import time from pathlib import Path import httpx def load rows kind, limit : lines = Path "data/prompts.jsonl" .read text .splitlines rows = json.loads line for line in lines rows = r for r in rows if kind == "all" or r "kind" == kind return rows :limit def pct values, p : ordered = sorted values return ordered min len ordered - 1, int p / 100 len ordered async def one client, url, model, row : body = { "model": model, "messages": row "messages" , "max tokens": row "max tokens" , "temperature": 0, "stream": True, "stream options": {"include usage": True}, } start = time.perf counter first = None pieces = tokens = 0 async with client.stream "POST", url, json=body as resp: resp.raise for status async for line in resp.aiter lines : if not line.startswith "data: " : continue payload = line 6: if payload == " DONE ": break event = json.loads payload if event.get "usage" : tokens = event "usage" "completion tokens" for choice in event.get "choices", : piece = choice.get "delta", {} .get "content" if piece: if first is None: first = time.perf counter pieces.append piece end = time.perf counter first = first or end decode s = max end - first, 1e-9 return { "id": row "id" , "ttft": first - start, "total": end - start, "tokens": tokens, "decode tok s": max tokens - 1, 0 / decode s, "text": "".join pieces , } async def run args : rows = load rows args.kind, args.limit url = f"http://localhost:{args.port}/v1/chat/completions" gate = asyncio.Semaphore args.concurrency async with httpx.AsyncClient timeout=600 as client: await one client, url, args.model, rows 0 warm-up, not recorded async def guarded row : async with gate: return await one client, url, args.model, row wall start = time.perf counter results = await asyncio.gather guarded r for r in rows wall = time.perf counter - wall start total tokens = sum r "tokens" for r in results summary = { "label": args.label, "kind": args.kind, "concurrency": args.concurrency, "requests": len results , "agg tok s": total tokens / wall, "p50 decode tok s": pct r "decode tok s" for r in results , 50 , "p50 ttft": pct r "ttft" for r in results , 50 , "p95 ttft": pct r "ttft" for r in results , 95 , } Path "results" .mkdir exist ok=True out = Path f"results/{args.label} {args.kind} c{args.concurrency}.json" out.write text json.dumps {"summary": summary, "rows": results}, indent=1 print f"{args.label:<9} {args.kind:<5} c={args.concurrency:<3} " f"agg={summary 'agg tok s' :.1f} tok/s " f"p50 decode={summary 'p50 decode tok s' :.1f} tok/s " f"p50 ttft={summary 'p50 ttft' 1000:.0f} ms" if name == " main ": ap = argparse.ArgumentParser ap.add argument "--label", required=True ap.add argument "--kind", default="all", choices= "all", "code", "chat", "json" ap.add argument "--concurrency", type=int, default=1 ap.add argument "--limit", type=int, default=32 ap.add argument "--port", type=int, default=8000 ap.add argument "--model", default="meta-llama/Llama-3.1-8B-Instruct" asyncio.run run ap.parse args Two more notes on the client. It sends one unrecorded warm-up request so CUDA graph capture and caches do not pollute the first sample, and it uses a semaphore to hold exactly the concurrency you asked for. The --limit default of 32 keeps one run short. Raise it to 100 or more when you want tighter confidence on a final measurement. Step 8: Measure the Baseline Twice Start the plain server in one terminal. cd ~/specdec-lab && source .venv/bin/activate ./serve.sh baseline Startup takes a while the first time because the engine loads weights and captures graphs. Wait until the log says the application has started, or poll the health endpoint from a second terminal. until curl -sf localhost:8000/health /dev/null; do sleep 3; done && echo healthy Now run the baseline at concurrency 1 on the code prompts, and run it twice. python bench.py --label baseline --kind code --concurrency 1 python bench.py --label baseline --kind code --concurrency 1 The line format looks like this. The values here are placeholders that show the layout, not measurements. baseline code c=1 agg=NNN.N tok/s p50 decode=NNN.N tok/s p50 ttft=NN ms baseline code c=1 agg=NNN.N tok/s p50 decode=NNN.N tok/s p50 ttft=NN ms The two runs should agree within a few percent. The gap between them is your noise floor, and any speculative speedup smaller than that gap is not real. If the two runs differ by 10 percent or more, something else is using the GPU, the card is power-throttling or the clocks are settling. Fix that first. A laptop GPU and a card in a hot case are the usual culprits. Check the server’s metrics endpoint once. In baseline mode it should contain no speculative decoding counters. curl -s localhost:8000/metrics | grep -c spec decode || true A count of 0 confirms that the baseline is clean. Stop the server with Ctrl+C before moving on, and confirm with nvidia-smi that memory has been released. Steps 9 to 11: Turn On N-Gram, Draft-Model and EAGLE3 Speculation Each method follows the same loop: start the server in its mode, wait for health, run the same benchmark command, read the acceptance counters, stop the server. Use the same three prompt kinds so the comparison stays fair. Step 9: N-gram lookup ./serve.sh ngram in a second terminal python bench.py --label ngram --kind code --concurrency 1 python bench.py --label ngram --kind chat --concurrency 1 curl -s localhost:8000/metrics | grep spec decode Predict the result before you read it. Code edits and JSON extraction copy heavily from the prompt, so n-gram lookup should do well there. Fresh chat answers should show a poor acceptance rate, and the speedup there should sit near 1.0x or slightly below. If chat comes out clearly faster, check that your prompts are not secretly repetitive. The metrics grep prints counters for drafted and accepted tokens. The exact metric names change between versions, so grep for spec decode and divide accepted by drafted yourself. An output shaped like the following is what you want to see. vllm:spec decode num drafts total{...} NNNN.0 vllm:spec decode num draft tokens total{...} NNNNN.0 vllm:spec decode num accepted tokens total{...} NNNNN.0 Step 10: A small draft model ./serve.sh draft in a second terminal python bench.py --label draft --kind chat --concurrency 1 python bench.py --label draft --kind code --concurrency 1 This mode loads two models, so startup is slower and memory use is higher. If it runs out of memory, shrink --max-model-len to 4096 in serve.sh or raise --gpu-memory-utilization a little, but change the same flag for every mode so the comparison stays honest. A same-family 1B draft typically agrees with its 8B sibling on easy, high-confidence tokens such as punctuation, common words and boilerplate, and disagrees on content choices. Expect a better acceptance rate than n-gram on chat, and a worse one than n-gram on code that copies the prompt verbatim. Step 11: EAGLE3 ./serve.sh eagle3 in a second terminal python bench.py --label eagle3 --kind chat --concurrency 1 python bench.py --label eagle3 --kind code --concurrency 1 EAGLE3 drafts with a head that reads the target’s own hidden states, so its proposals track the target more closely than a separate small model does. That is the reason it often posts the highest acceptance rate on open-ended text. It also depends entirely on the head matching the target and the chat template. If acceptance sits near zero, you probably loaded a head trained for a different model or a different instruction format. The troubleshooting table covers it. Step 12: Sweep Concurrency and Verify Lossless Output Running each mode by hand works for one data point. A sweep across prompt kinds and concurrency levels 1, 4 and 16 is what exposes the crossover where speculation stops paying. The runner script automates the whole matrix: it starts each server, waits for health, runs nine benchmarks, saves the speculative counters and stops the server. bash /usr/bin/env bash set -euo pipefail cd "$ dirname "$0" " mkdir -p logs results data -f data/prompts.jsonl || python make prompts.py for MODE in baseline ngram draft eagle3; do echo "== starting $MODE" ./serve.sh "$MODE" "logs/server $MODE.log" 2 &1 & PID=$ until curl -sf localhost:8000/health /dev/null; do kill -0 "$PID" 2 /dev/null || { echo "server for $MODE died, read logs/server $MODE.log"; exit 1; } sleep 5 done for KIND in code chat json; do for C in 1 4 16; do python bench.py --label "$MODE" --kind "$KIND" --concurrency "$C" done done curl -s localhost:8000/metrics | grep spec decode "results/metrics $MODE.txt" || true kill "$PID" wait "$PID" 2 /dev/null || true sleep 10 done python compare.py | tee results/summary.txt The comparison script then reads the saved results and prints one row per method, kind and concurrency level, with speedup relative to the baseline at the same settings and a count of requests whose text matches the baseline exactly. python import json from pathlib import Path LABELS = "baseline", "ngram", "draft", "eagle3" KINDS = "code", "chat", "json" CONCURRENCY = 1, 4, 16 def load label, kind, conc : path = Path f"results/{label} {kind} c{conc}.json" return json.loads path.read text if path.exists else None print f"{'kind':<6}{'conc': 5} {'method':<9}{'tok/s': 9}{'speedup': 9}{'p50 ttft ms': 13}{'same text': 11}" for kind in KINDS: for conc in CONCURRENCY: base = load "baseline", kind, conc if not base: continue base rate = base "summary" "agg tok s" for label in LABELS: run = load label, kind, conc if not run: continue s = run "summary" same = sum 1 for a, b in zip base "rows" , run "rows" if a "text" == b "text" total = len base "rows" print f"{kind:<6}{conc: 5} {label:<9}{s 'agg tok s' : 9.1f}" f"{s 'agg tok s' / base rate: 8.2f}x{s 'p50 ttft' 1000: 13.0f}{same: 7}/{total}" Run everything with one command. ./run all.sh The output has this shape. Again, the figures are placeholders that show the layout. kind conc method tok/s speedup p50 ttft ms same text code 1 baseline NNN.N 1.00x NN 32/32 code 1 ngram NNN.N N.NNx NN NN/32 code 1 draft NNN.N N.NNx NN NN/32 code 1 eagle3 NNN.N N.NNx NN NN/32 code 4 baseline NNN.N 1.00x NN 32/32 ... How to read it. Start with the identical-text column. If a method matches the baseline on nearly every request, the lossless property is holding in practice. A handful of mismatches at higher concurrency is normal, because batch composition changes floating-point reduction order and can flip a near-tie. If most of the text differs, treat that as a bug in your configuration rather than a feature. Then read the speedup column per concurrency level. Most setups show the largest gains at concurrency 1 and a shrinking or negative gain at 16. Turn the table into a capacity decision. A per-user speedup is not a capacity gain. If a method raises P50 decode speed at concurrency 1 but leaves aggregate tok/s flat at concurrency 16, you bought a better experience for quiet hours and nothing for peak load. Write both numbers down. If your peak concurrency sits in the low single digits, trust the per-user column. If you run dozens of streams per GPU, trust the aggregate column and be ready to find that plain decoding wins. Check acceptance before you tune anything else. When a method underperforms, divide accepted draft tokens by proposed draft tokens using the saved metrics files. An acceptance rate below roughly one half usually means the proposer does not fit the workload, and shortening the draft will not rescue it. A rate above about three quarters with a disappointing speedup points at overhead instead, such as an expensive draft, too little spare GPU capacity or a long context that makes the draft’s own cache heavy. Use the table below as a decision aid once you have your own numbers. The middle column is what the mechanics predict, so it is a hypothesis you are testing, not a result. | Workload | Method to try first | What to check in your results | |---|---|---| | Code rewriting, refactors, edits | N-gram or suffix | Acceptance rate on code prompts and the speedup at concurrency 1 and 4 | | Structured extraction to JSON | N-gram | Whether short outputs leave enough decode time to matter at all | | Open-ended chat at low concurrency | EAGLE3 or a draft model | Speedup in P50 decode tok/s with TTFT unchanged | | Busy API at high concurrency | Often no speculation | Aggregate tok/s at concurrency 16 and above versus the baseline | | Mixed traffic | Measure per route | Whether one config helps every kind or hurts one of them | If you need to prove to someone else that the model you serve is the model you claim, speculation is a point to disclose. Output equality at temperature 0 is evidence, not proof, and provable-inference research such as FlashProve https://shattered.io/anthropic-flashprove-provable-inference-72x-overhead-2026/ targets exactly the gap between a claimed model and the one actually serving tokens. For ordinary serving, the identical-output count is enough. Beyond Draft Models: MTP Heads and llama.cpp for Single-User Setups MTP in vLLM. If your target model ships with native multi-token-prediction heads, you do not need a second checkpoint. The vLLM configuration is a single JSON object with the method set to mtp and one speculative token. vllm serve YOUR TARGET WITH MTP HEADS \ --speculative-config '{"method":"mtp","num speculative tokens":1}' vLLM’s own example uses XiaomiMiMo/MiMo-7B-Base. Google also released MTP drafters for the Gemma 4 family under the Apache 2.0 license according to its announcement, which positions them as usable across Transformers, MLX, vLLM, SGLang, Ollama and Google AI Edge Gallery. To add MTP to the lab, add a fifth case to serve.sh and pass the new target through the TARGET variable, but remember that the identical-text comparison only makes sense against a baseline of the same target. Other engines. SGLang has its own speculative decoding documentation https://docs.sglang.io/docs/advanced features/speculative decoding with a similar set of methods, and NVIDIA’s NIM containers expose speculation too. One trap there: NIM for LLMs 2.1.5 updated its vLLM backend to 0.31.0, so the container’s flags and features trail what you installed from PyPI at 0.33.0. Read the container’s own docs, not the upstream ones. llama.cpp for a single GPU or CPU box. If you run a local assistant on a desktop and care about one user’s speed, llama.cpp is a natural fit. It supports running a main model with a smaller draft model in llama-server. The flags below are the commonly used ones, but they shift between builds, so confirm them against llama-server --help on your build. The project reportedly marked v0.9.0 as its latest stable tag on November 4, 2026, with development builds publishing almost daily after that. llama-server \ -m models/target-Q4 K M.gguf \ -md models/draft-Q8 0.gguf \ --draft-max 16 --draft-min 1 \ -ngl 99 -ngld 99 -c 8192 --port 8080 The same measurement discipline applies. Run the same prompts with and without the -md flag, compare per-request decode speed and check that the output text is unchanged at temperature 0. Point the --port option of the benchmark script at your running llama-server instance. Common Pitfalls That Ruin Speculative Decoding Benchmarks - Counting stream chunks as tokens. Speculation delivers several tokens per chunk. Read the usage block, as the client does, or your speculative runs will look slower than the baseline. - Testing at one concurrency only. A method that wins at concurrency 1 can lose at 16. Report the sweep, not a single flattering row. - Skipping the noise floor. Run the baseline twice. A 3 percent gain inside a 5 percent wobble is not a result. - Using a draft with a different tokenizer. The verification step compares token IDs. A mismatched vocabulary either errors out or produces near-zero acceptance. Use the same family. - Setting the speculative token count too high. More proposed tokens means more wasted work when acceptance is mediocre. Start at 3 to 5 and sweep. - Following a Medusa or MLP speculator guide. Those paths were reportedly removed in vLLM 0.33.0. Check the date on any guide you follow. - Expecting identical text with sampling on. Temperature above 0 changes the text run to run by design. Keep temperature at 0 for the equality check. - Letting the draft starve the KV cache. Draft weights and the draft’s own cache come out of the same VRAM budget. Long contexts plus a draft is where out-of-memory errors appear. - Comparing against a warm server. Prefix caching can make a second run faster for reasons unrelated to speculation. The runner restarts the server per method, which avoids that. - Trusting a vendor speedup claim for your stack. A claim like Google’s up to 3x is measured on specific hardware with specific prompts. Reproduce it or discount it. Troubleshooting: 12 Problems and Fixes | Symptom | Likely cause | Fix | |---|---|---| | 401 or 403 when downloading a Llama checkpoint | Gated repo or missing token | Accept the license on the model page, create a read token and export HF TOKEN | | Import error mentioning a CUDA library right after install | Driver too old for the wheel | Update the NVIDIA driver, or install a vLLM build that matches your CUDA stack. Note that 0.33.0 dropped CUDA 12.9 | | Error parsing speculative config | JSON quoting broken by the shell | Wrap the JSON in single quotes, or escape inner double quotes as the script does | | Unknown speculative method | vLLM older than 0.33.0 | Run vllm --version , then upgrade to 0.33.0 inside the venv | | Vocabulary or tokenizer mismatch error with a draft | Draft from a different model family | Choose a draft that shares the target’s tokenizer | | Out-of-memory at startup in draft mode | Draft weights plus cache exceed the budget | Lower --max-model-len or use a quantized or smaller draft, and keep the flag identical across modes | | Speculative run slower than baseline | Low acceptance or high concurrency | Lower num speculative tokens , try n-gram on copy-heavy prompts, or test at lower concurrency | | Outputs differ from the baseline at temperature 0 | Numeric drift from batching | Count how many differ and where they diverge. A few late divergences are normal, while early divergence in most requests is a config bug | | No spec decode lines in the metrics | Baseline mode, or renamed counters | Confirm the mode, then grep for spec without the suffix | | EAGLE3 acceptance near zero | Head trained for a different target or template | Use the head published for your exact target and keep the chat template identical | | Server never reports healthy | Still downloading, loading or capturing graphs | Read logs/server MODE.log , wait out the first start and pre-download checkpoints in Step 4 | | GPU memory still used after stopping | A child process survived the kill | Run pkill -f 'vllm serve' and check nvidia-smi before the next mode | Advanced Tips for Tuning and Training Your Own Drafters Sweep the draft length. The best number of speculative tokens depends on the workload. Add a loop around serve.sh that tries 2, 3, 4, 6 and 8 for each method and plot P50 decode tok/s against it. Copy-heavy workloads usually tolerate longer drafts, while free-form chat peaks earlier. Train a drafter on your own traffic. A generic draft head matches a generic distribution. If your product produces a narrow kind of text, a drafter trained on that text raises acceptance. vLLM’s Speculators project is documented as a library for training draft models that integrate with vLLM, according to the project’s docs. NVIDIA’s NeMo AutoModel documentation, according to its EAGLE-3 training recipe page https://docs.nvidia.com/nemo/automodel/latest/nemo-automodel/nemo automodel/recipes/speculative decoding/train eagle3 , covers EAGLE-3 training for Llama-style dense models and Qwen3-style mixture-of-experts backbones. On AMD hardware, AMD’s ROCm blog describes training EAGLE3 draft models on Instinct MI350X https://rocm.blogs.amd.com/artificial-intelligence/eagle3-training/README.html with Quark 0.13, dated October 27, 2026. Fine-tune https://shattered.io/fine-tune-llm-qlora-12-steps-2026/ built-in MTP heads. NVIDIA’s NeMo documentation reportedly covers fine-tuning a model’s native MTP heads for domain-specific self-speculative decoding. That path keeps a single checkpoint, which simplifies deployment. Know which verification modes are lossy. The 0.33.0 notes reportedly add an opt-in lossy verification policy and refiners for block-diffusion drafters. Those can raise throughput by accepting tokens the target would not have chosen. Keep them off for evals, for anything with compliance requirements and for any run where you want the identical-text count to mean something. Switch speculation off under load. Since the gain shrinks as batches grow, a production router can send latency-sensitive interactive traffic to a speculative deployment and bulk traffic to a plain one. Measure the crossover in your own sweep, then encode it as a routing rule instead of guessing. Test with your real output lengths. Speculation pays off per decoded token, so short answers leave little to save. A JSON extraction that emits 40 tokens spends most of its wall time on prompt processing, and no draft method changes that. Keep the token caps in make prompts.py close to what production sends. Watch long contexts. Verification reads the whole KV cache for every candidate, and a draft model keeps its own cache. If your product runs long documents through the model, rerun the sweep at that context length. A configuration that wins at 2,000 tokens of context can lose at 30,000. Export acceptance rate to your dashboards. A drop in acceptance usually means the prompt mix shifted, a model got swapped or a checkpoint is mismatched. It is a cheap early-warning signal, and the counters already exist on the metrics endpoint. Keep the lab around. Re-run ./run all.sh whenever you upgrade vLLM, swap the target or change GPUs. Results from one stack do not transfer to the next. More walkthroughs live in the AI and machine learning section https://shattered.io/category/ai-machine-learning/ for the model-side work that feeds a harness like this one. The Complete Working Project The finished lab is eight small files. Nothing outside this layout is required. specdec-lab/ requirements.txt vllm==0.33.0 and httpx make prompts.py builds data/prompts.jsonl 112 prompts, 3 kinds serve.sh baseline | ngram | draft | eagle3 bench.py streaming client, writes results/ .json compare.py speedup and identical-text table run all.sh runs the full matrix and saves summary.txt data/ results/ logs/ The end-to-end sequence from an empty machine looks like this. mkdir -p ~/specdec-lab && cd ~/specdec-lab uv venv --python 3.12 .venv && source .venv/bin/activate uv pip install -r requirements.txt export HF TOKEN=hf your read token here hf download meta-llama/Llama-3.1-8B-Instruct hf download meta-llama/Llama-3.2-1B-Instruct hf download vllm-project/EAGLE3-LLaMA3.1-Instruct-8B python make prompts.py ./run all.sh cat results/summary.txt When it finishes, results/summary.txt holds the comparison and results/metrics .txt holds the speculative counters for each mode. Keep both next to the vLLM version and driver version you ran, because a speedup number without its stack is not worth saving. Write down the decision in one sentence per workload, for example which method you will deploy for interactive chat and which route stays plain. A final thought on scope. Speculative decoding is a latency tool for memory-bound serving, and the honest answer for many deployments is that it helps one route and does nothing for another. The lab exists so that answer comes from your GPU and your prompts, not from somebody else’s chart. FAQ What is speculative decoding in simple terms? A small, fast proposer guesses the next several tokens and the large model checks all of them in one pass. Accepted guesses are free speed, and rejected ones are discarded so the final text matches what the large model would have written alone. Does speculative decoding change the model’s output? In greedy mode, accepted tokens must equal the target’s own choice, so the text should be the same as without speculation. With sampling, standard rejection-sampling rules preserve the target distribution. Floating-point differences can still flip rare near-ties, which is why the lab counts identical outputs. Opt-in lossy verification modes are the exception and should stay off for evals. How much faster will my model get? It depends on acceptance rate, draft cost, GPU and concurrency. Google claims up to 3x for its Gemma 4 drafters, and the back-of-envelope math in the first section gives a ceiling of roughly 2.4x for an 80 percent acceptance rate with a cheap draft. Your measurement from Step 12 is the only number that counts for your stack. Which method should I try first? For copy-heavy work such as code edits and structured extraction, start with n-gram lookup because it needs no extra model. For open-ended chat at low concurrency, try EAGLE3 if a head exists for your target, or a same-family draft model. If your model ships native MTP heads, try those before downloading anything. Why did speculation make my server slower? Most often the batch is large enough that decode is no longer memory-bound, so verifying extra tokens costs real compute. Low acceptance is the second cause. Lower the draft length, test at lower concurrency and compare acceptance rates before blaming the engine. Can I use a draft model from a different family? Not reliably. The verifier compares token IDs, so the draft needs the same tokenizer as the target. Models from the same family usually qualify. A different family either fails at startup or produces acceptance near zero. Does speculative decoding need more GPU memory? Yes for draft models and EAGLE heads, because their weights and cache live next to the target’s. N-gram lookup adds almost none. Plan for the extra footprint and test with your real context length, since long contexts are where out-of-memory errors show up first. Is this worth doing on a local desktop? For a single-user assistant, usually yes, because single-stream decode is exactly where speculation helps most. Try llama.cpp’s draft-model flags or vLLM’s n-gram mode, run the same prompts both ways and keep whichever gives a better per-request decode speed with unchanged text.