{"slug": "speculative-decoding-tutorial-12-steps-75-min-2026", "title": "Speculative Decoding Tutorial: 12 Steps, 75 Min [2026]", "summary": "A tutorial published on Shattered.io lays out a 12-step, roughly 75-minute harness for measuring speculative decoding speedups across four configurations — a plain baseline, n-gram lookup, a small draft model and EAGLE3 — on the reader's own hardware. The tutorial cites Google's claim of up to a 3x inference-speed improvement from Gemma 4 multi-token-prediction drafters without quality degradation, while noting that figure is a vendor claim and that real gains depend on GPU, prompt mix and concurrency. It gives the acceptance-rate formula (1 – a^(k+1)) / (1 – a), which yields about 3.4 tokens per target pass at a = 0.8 and k = 4 versus about 1.9 at a = 0.5, and warns that speculative decoding helps memory-bound low-QPS workloads but often yields no gain for a saturated multi-tenant API.", "body_md": "Don't miss new tech stories on Google\n\nAdd Shattered once in the Google app and our stories show up in your news suggestions.\n\n[Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F)\n\n## How Speculative Decoding Works and When It Pays Off\n\nSpeculative decoding lets a cheap proposer guess several tokens ahead while the full model checks every guess in a single forward pass, so the text stays the same and the number of expensive passes drops. Google claims [up to a 3x inference-speed improvement](https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/) from its Gemma 4 multi-token-prediction drafters without degradation in output quality or reasoning logic. That is a vendor claim, and your own number will land somewhere below it or occasionally above it depending on GPU, prompt mix and concurrency. The 12 steps below build a small lab that measures the real speedup on your hardware, in about 75 minutes, with four configurations: a plain baseline, n-gram lookup, a small draft model and EAGLE3.\n\nThe reason it works is simple. Generating one token with an 8B-parameter model means reading all of its weights from GPU memory, and the arithmetic units sit mostly idle while that happens. Checking five candidate tokens in the same pass costs only slightly more than checking one, because the weights are read once either way. If the proposer guessed right, you bank several tokens for the price of one step. If it guessed wrong, you keep the tokens up to the first mistake and lose a little time on the rest.\n\nThe open-weight field keeps moving, and the [best open source LLM](https://shattered.io/best-open-source-llm-october-2026-kimi-glm-deepseek/) options in late 2026 increasingly ship with draft heads or matching drafters. That makes speculative decoding a default thing to evaluate, not an exotic trick. The catch is that every serving stack names the knobs differently, the docs lag the code, and most tutorials online show numbers from a different GPU, a different prompt set and a different library version. A repeatable harness fixes that.\n\n**The acceptance math you should keep in your head.** Call the chance that the target model accepts any single drafted token the acceptance rate, written as a. If the draft proposes k tokens per step and each is accepted independently with probability a, the expected number of tokens produced per target pass is (1 – a^(k+1)) / (1 – a). With a = 0.8 and k = 4, that works out to about 3.4 tokens per pass. With a = 0.5 and k = 4, it falls to about 1.9. The drafter is not free, so divide by the cost of the extra work. If one draft step costs 10 percent of a target pass, then k = 4 adds 0.4 of a pass, and the back-of-envelope ceiling at a = 0.8 is about 3.4 divided by 1.4, or about 2.4x. Real systems land below that because verification adds overhead and acceptance is not independent, but the shape holds: acceptance rate dominates everything.\n\n**When it helps and when it hurts.** Decode is memory-bound at small batch sizes, which is where speculation shines. As you pile on concurrent requests, the GPU turns compute-bound, the spare capacity that verification relied on disappears, and rejected tokens become pure waste. The vLLM speculative decoding guide reportedly frames the technique as a way to cut inter-token latency in medium-to-low-QPS, memory-bound workloads, and that matches what the sweep in Step 12 usually shows. A single-user local assistant benefits most. A saturated multi-tenant API often sees no gain at all.\n\n**The lossless property.** In greedy mode (temperature 0), a drafted token is accepted only if it equals the token the target would have picked, so the final text should match a non-speculative run. With sampling, vLLM and similar engines use a rejection-sampling rule that preserves the target distribution. Small numeric differences from batching can still flip a near-tie, so the harness counts identical outputs instead of assuming all of them match. Some stacks also expose opt-in lossy verification modes (more on that in the advanced tips), and those break the guarantee on purpose.\n\n**A worked example before you spend GPU time.** The numbers here are hypothetical and exist only to show the reasoning. Suppose a chat reply averages 250 output tokens and one user sees 40 tokens per second without speculation, so a reply takes about 6.3 seconds. A 1.8x per-user gain cuts that to roughly 3.5 seconds, which people notice. Now take a nightly batch job that nobody watches. The same 1.8x gain at a single stream means little there, because the job cares about aggregate throughput at full batch, and that is exactly where speculation tends to fade. The same technique can be a clear win for one product and irrelevant for another, which is why the lab measures both views.\n\n## Speculative Decoding Methods in vLLM v0.33.0 Compared\n\nvLLM’s documentation lists model-based methods (EAGLE, MTP and draft models) plus n-gram and suffix decoding, all configured through a single `--speculative-config` JSON argument. The table summarizes how they differ in what you must download and when they tend to win. The last column is analysis based on how each method works, not a measured result, so treat it as a hypothesis for the lab to confirm.\n\n| Method | Separate draft checkpoint | Training needed | Likely sweet spot | Config key | \n|---|---|---|---|---|\n| Draft model | Yes, a small model with the same tokenizer | No, if a matching small sibling exists | General chat when a same-family small model exists | `draft_model` | \n| N-gram lookup | No | No | Code edits, JSON extraction, summaries that copy the prompt | `ngram` | \n| Suffix decoding | No | No | Agent loops that repeat earlier tool output | `suffix` | \n| EAGLE3 | Yes, a head trained for your exact target | Yes, unless a published head exists | Open-ended chat at low concurrency | `eagle3` | \n| MTP | No, uses heads built into the target | Built in by the model maker | Models that ship with native prediction heads | `mtp` | \n| Medusa and MLP speculator | Yes | Yes | Legacy setups only | Removed (reportedly) in 0.33.0 | \n\n**Draft model.** A small sibling of your target proposes tokens, and the big model verifies them. The draft must share the target’s tokenizer, which in practice means the same model family. According to vLLM’s release notes, draft-model speculation reportedly landed on the newer Model Runner V2 in v0.31.0. It is the most intuitive method and the easiest to reason about, but it costs extra VRAM for the draft weights and its own KV cache.\n\n**N-gram lookup.** No model at all. The engine looks at the last few tokens, searches the prompt for the same sequence and proposes whatever followed it there. When the output copies from the input (rewriting code, extracting fields from a paragraph, quoting a document) acceptance can be very high at almost zero cost. When the output is fresh prose, it proposes nothing useful, and you pay a small lookup tax for nothing.\n\n**Suffix decoding.** A cousin of n-gram lookup that matches against earlier generated text and, in recent vLLM builds, reportedly against a cross-request memory on the GPU. It targets repetitive agent workloads. The lab skips it because its benefit depends on the traffic pattern, but the harness makes it a one-line addition.\n\n**EAGLE3.** A small trained head that reads the target’s hidden states to propose tokens. NVIDIA describes EAGLE3 as a separate draft model that proposes tokens for the larger target to verify, which means you need a published checkpoint that matches your target. The lab uses the [vllm-project copy of the EAGLE3 Llama 3.1 8B Instruct checkpoint](https://huggingface.co/vllm-project/EAGLE3-LLaMA3.1-Instruct-8B/blob/main/README.md). Its model card says the copy mirrors the yuhuili original at a specific commit, made on 2026-10-30, because vLLM’s CI uses it and the original lives on a personal account. That is a supply-chain decision worth copying in your own pipelines.\n\n**MTP.** Multi-token prediction uses extra heads that the model maker trained into the target itself, so there is no second download. In vLLM it is enabled with the method set to mtp and `num_speculative_tokens` set to 1. It only works on models that actually contain those heads. A model without them needs EAGLE or a draft model instead.\n\n**Medusa and MLP speculators.** Older tutorials still show these. The 0.33.0 release notes reportedly removed both, along with the legacy Ray compiled-graph executor. If a guide tells you to launch with a Medusa head, check its date before you spend an evening on it.\n\n**What changed in the last quarter.** Three shifts matter for anyone working from an older guide. First, vLLM v0.33.0 is the current line, built on PyTorch 2.14, and its notes reportedly add refiners for block-diffusion drafters such as DFlash and DSpark, drop Medusa and the MLP speculator and carry an opt-in lossy verification policy. Second, vendors now publish training paths for drafters and not only finished checkpoints. AMD’s ROCm team described EAGLE3 training on Instinct MI350X on October 27, 2026, and NVIDIA’s NeMo AutoModel documentation carries an EAGLE-3 recipe. Third, model makers increasingly ship the drafter alongside the model, as Google did for Gemma 4. The practical effect is that a draft path you ruled out earlier this year may exist today, so re-test instead of relying on memory.\n\n## Prerequisites and Versions\n\nThe lab targets a single Linux box with one NVIDIA GPU. Everything below was chosen so that each component has an official source for its current state as of November 13, 2026.\n\n| Component | Version used | Notes | \n|---|---|---|\n| GPU | NVIDIA, 24 GB of VRAM or more | Weights for an 8B model at 16-bit precision take roughly 16 GB before KV cache. GeForce RTX 5090 is NVIDIA’s current desktop flagship, and any card with enough memory works. | \n| NVIDIA driver | A current production driver | Install the driver on its own. PyTorch wheels bring their own CUDA runtime, so a full CUDA toolkit is optional for this lab. | \n| Python | 3.12 | vLLM requires 3.10 or newer and recommends 3.12 or newer. | \n| vLLM | 0.33.0 | The current release line. It moved to PyTorch 2.14 and dropped CUDA 12.9 support, and it is compatible with CUDA 13.x. | \n| PyTorch | 2.14 (installed by vLLM) | Do not pin it yourself. Let the vLLM wheel pick it. | \n| httpx | Latest | Async HTTP client for the benchmark script. | \n| Target model | meta-llama/Llama-3.1-8B-Instruct | Gated on Hugging Face. Accept the license first. | \n| Draft model | meta-llama/Llama-3.2-1B-Instruct | Same tokenizer family as the target. Also gated. | \n| EAGLE3 head | vllm-project/EAGLE3-LLaMA3.1-Instruct-8B | Trained for the exact Llama 3.1 8B Instruct target. | \n| uv | Latest | Fast environment manager, optional but used below. | \n\nTwo honest caveats. First, Llama 3.1 8B Instruct is not a new model, but it is the best-documented target for a three-way comparison because a published EAGLE3 head, a same-tokenizer small sibling and vLLM’s own CI all exist for it. The harness is model-agnostic through three environment variables, so you can swap in a newer target once you have a matching drafter. Second, plan for about 30 GB of free disk for the three checkpoints and the Hugging Face cache, and a Hugging Face token with access to the gated Llama repositories.\n\nRecord the stack before you run anything. Save the output of `nvidia-smi`, `vllm --version` and `uv pip freeze` next to the results, because a speedup without its stack attached cannot be reproduced and will mislead whoever reads it six months from now. The inference ecosystem moved quickly in the last quarter, with a new vLLM release line, new drafter checkpoints and changed defaults, so a result is only as good as the versions behind it.\n\n## Steps 1 to 3: Define Metrics, Check the GPU, Build the Environment\n\n**Step 1: Fix the metrics before you touch the GPU.** A benchmark that changes its definition halfway through is useless. The lab records five numbers per run, and the table says what each one answers.\n\n| Metric | How the harness computes it | What it tells you | \n|---|---|---|\n| Aggregate tok/s | Total completion tokens divided by wall-clock time for the whole batch | Throughput. This is the number a busy server cares about. | \n| P50 decode tok/s | Per request, (tokens minus 1) divided by (end minus first token time), then the median | Per-user speed. This is what one person typing into a chat feels. | \n| P50 and P95 TTFT | Time from request start to first content chunk | Whether speculation hurt the first-token delay. It should not move much. | \n| Acceptance rate | Accepted draft tokens divided by proposed draft tokens, from the server’s metrics | Why a method won or lost. | \n| Identical outputs | Count of requests whose text equals the baseline text at temperature 0 | Whether the lossless property held in practice. | \n\nDecide now what counts as a win. A reasonable rule: a method earns its place if it improves P50 decode tok/s by a clear margin at the concurrency you actually run, without lowering aggregate throughput at your peak load, and with the identical-output count staying near the full set.\n\n**Step 2: Check the GPU and driver.** Confirm the card is visible, note its memory, and make sure nothing else is holding VRAM.\n\n```\nnvidia-smi --query-gpu=name,memory.total,memory.used,driver_version --format=csv\n```\n\nThe output should list your GPU name, total memory, a near-zero used figure and the driver version. If another process holds several gigabytes, stop it now. A draft model and its cache need headroom, and a half-full GPU produces confusing out-of-memory errors in Step 10.\n\n**Step 3: Create the project and install vLLM 0.33.0.** The environment uses uv with Python 3.12. If you prefer pip and venv, the same package names apply.\n\n```\nmkdir -p ~/specdec-lab/{data,results,logs} && cd ~/specdec-lab\ncurl -LsSf https://astral.sh/uv/install.sh | sh\nuv venv --python 3.12 .venv\nsource .venv/bin/activate\nprintf 'vllm==0.33.0\\nhttpx\\n' > requirements.txt\nuv pip install -r requirements.txt\npython -c \"import vllm, torch; print('vllm', vllm.__version__, 'torch', torch.__version__)\"\n```\n\nExpected output on the last line:\n\n```\nvllm 0.33.0 torch 2.14.<patch>\n```\n\nYour PyTorch patch number may differ. What matters is vllm 0.33.0 and a torch 2.14 build. If the import fails with a CUDA library error, jump to the troubleshooting table, because the usual cause is a driver that is too old for the wheel.\n\n## Steps 4 and 5: Fetch Checkpoints and Build the Prompt Sets\n\n**Step 4: Download the three checkpoints.** Log in with a read-only token that has access to the Llama repositories, then pull the target, the small draft and the EAGLE3 head. Pre-downloading matters because vLLM would otherwise download during server start, and your health-check loop would time out on a slow connection.\n\n```\nexport HF_TOKEN=hf_your_read_token_here\nhf download meta-llama/Llama-3.1-8B-Instruct\nhf download meta-llama/Llama-3.2-1B-Instruct\nhf download vllm-project/EAGLE3-LLaMA3.1-Instruct-8B\ndu -sh ~/.cache/huggingface\n```\n\nThe Hugging Face command-line tool is called hf in current releases. If your install still exposes the older huggingface-cli name, the subcommand and arguments are the same. The last command prints the cache size so you can see that the three checkpoints landed.\n\n**Step 5: Generate three prompt workloads with different copy behavior.** Speculation lives or dies on how predictable the output is, so one prompt set would hide the story. The generator builds three kinds. Code-edit prompts ask the model to rewrite a short Python function, which means much of the answer repeats the input. Chat prompts ask for fresh explanations, which is the hard case for n-gram lookup. JSON extraction prompts pull fields out of a sentence, which is short and heavily copied. A fixed random seed keeps the set identical across machines.\n\n``` python\nimport json\nimport random\nfrom pathlib import Path\n\nrandom.seed(7)\nOUT = Path(\"data/prompts.jsonl\")\nOUT.parent.mkdir(exist_ok=True)\n\nFUNCS = [\n    \"def slugify(text):\\n    text = text.strip().lower()\\n    out = []\\n    for ch in text:\\n        out.append(ch if ch.isalnum() else '-')\\n    return ''.join(out)\\n\",\n    \"def mean(values):\\n    total = 0\\n    for v in values:\\n        total += v\\n    return total / len(values)\\n\",\n    \"def chunk(items, size):\\n    result = []\\n    for i in range(0, len(items), size):\\n        result.append(items[i:i + size])\\n    return result\\n\",\n    \"def dedupe(items):\\n    seen = set()\\n    out = []\\n    for x in items:\\n        if x not in seen:\\n            seen.add(x)\\n            out.append(x)\\n    return out\\n\",\n    \"def word_count(text):\\n    counts = {}\\n    for w in text.split():\\n        counts[w] = counts.get(w, 0) + 1\\n    return counts\\n\",\n    \"def clamp(x, lo, hi):\\n    if x < lo:\\n        return lo\\n    if x > hi:\\n        return hi\\n    return x\\n\",\n]\nTASKS = [\n    \"Add type hints and a docstring.\",\n    \"Add input validation that raises ValueError.\",\n    \"Rewrite it with a comprehension where that is cleaner.\",\n    \"Add logging calls at the start and the end.\",\n    \"Add three pytest tests below the function.\",\n    \"Rename the variables so each name says what it holds.\",\n]\nTOPICS = [\n    \"database indexes\", \"TCP congestion control\", \"CI caching\", \"feature flags\",\n    \"rate limiting\", \"idempotent APIs\", \"container image layers\", \"DNS caching\",\n    \"retry with backoff\", \"zero-downtime deploys\", \"observability budgets\",\n    \"schema migrations\", \"message queues\", \"blue-green releases\", \"secrets rotation\",\n    \"load balancer health checks\", \"log sampling\", \"connection pooling\",\n    \"canary analysis\", \"dependency pinning\",\n]\nNAMES = [\"Ana Ruiz\", \"Ben Okafor\", \"Chloe Martin\", \"Dev Patel\", \"Elif Demir\", \"Farid Haddad\"]\nCITIES = [\"Lisbon\", \"Austin\", \"Leeds\", \"Pune\", \"Gdansk\", \"Denver\", \"Osaka\", \"Tallinn\"]\n\nrows = []\nfor i, (fn, task) in enumerate((f, t) for f in FUNCS for t in TASKS):\n    prompt = f\"Here is a Python function:\\n\\n{fn}\\n{task} Return the full updated code in one block.\"\n    rows.append({\"id\": f\"code-{i:03d}\", \"kind\": \"code\", \"max_tokens\": 400,\n                 \"messages\": [{\"role\": \"user\", \"content\": prompt}]})\nfor i, topic in enumerate(TOPICS):\n    for j, tpl in enumerate([\n        \"Explain {t} to a new engineer in about 200 words.\",\n        \"Give three pitfalls of {t}, one sentence each, then one fix per pitfall.\",\n    ]):\n        rows.append({\"id\": f\"chat-{i:03d}-{j}\", \"kind\": \"chat\", \"max_tokens\": 300,\n                     \"messages\": [{\"role\": \"user\", \"content\": tpl.format(t=topic)}]})\nfor i in range(36):\n    text = (f\"Order {1000 + i * 7} was placed by {random.choice(NAMES)} from \"\n            f\"{random.choice(CITIES)} on 2026-{random.randint(1, 11):02d}-{random.randint(1, 28):02d} \"\n            f\"for {random.randint(20, 900)}.{random.randint(0, 99):02d} USD.\")\n    prompt = (\"Extract order_id, customer, city, date and amount as one JSON object. \"\n              f\"Return JSON only.\\n\\n{text}\")\n    rows.append({\"id\": f\"json-{i:03d}\", \"kind\": \"json\", \"max_tokens\": 120,\n                 \"messages\": [{\"role\": \"user\", \"content\": prompt}]})\n\nOUT.write_text(\"\\n\".join(json.dumps(r) for r in rows) + \"\\n\")\nprint(f\"wrote {len(rows)} prompts to {OUT}\")\nfor kind in (\"code\", \"chat\", \"json\"):\n    print(kind, sum(1 for r in rows if r[\"kind\"] == kind))\n```\n\nRun it once.\n\n```\npython make_prompts.py\nwrote 112 prompts to data/prompts.jsonl\ncode 36\nchat 40\njson 36\n```\n\nSynthetic prompts are a starting point, not the finish line. If you serve real traffic, replace `data/prompts.jsonl` with a sample of it. Public request logs help here, such as the [6.12 billion request LLM dataset from Chutes and Harvard](https://shattered.io/chutes-harvard-6-12-billion-llm-dataset-2026/), which gives you a sense of how varied real prompt and output lengths are. The benchmark client only needs the same three fields per row: an id, a kind, a messages list and a token cap.\n\n## Steps 6 and 7: Write the Server Launcher and the Streaming Benchmark Client\n\n**Step 6: One launcher script, four modes.** The script maps a mode name to a `--speculative-config` JSON string, so every run uses identical server flags apart from the speculation setting. It fixes the seed, caps the context at 8192 tokens and leaves 10 percent of VRAM free. The environment variables at the top let you swap the target, the draft or the EAGLE head without editing the file.\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\n\nMODE=\"${1:-baseline}\"\nTARGET=\"${TARGET:-meta-llama/Llama-3.1-8B-Instruct}\"\nDRAFT=\"${DRAFT:-meta-llama/Llama-3.2-1B-Instruct}\"\nEAGLE=\"${EAGLE:-vllm-project/EAGLE3-LLaMA3.1-Instruct-8B}\"\nPORT=\"${PORT:-8000}\"\n\nCOMMON=(--port \"$PORT\" --max-model-len 8192 --gpu-memory-utilization 0.90 --seed 0)\n\ncase \"$MODE\" in\n  baseline) EXTRA=() ;;\n  ngram)  EXTRA=(--speculative-config '{\"method\":\"ngram\",\"num_speculative_tokens\":5,\"prompt_lookup_max\":4}') ;;\n  draft)  EXTRA=(--speculative-config \"{\\\"method\\\":\\\"draft_model\\\",\\\"model\\\":\\\"$DRAFT\\\",\\\"num_speculative_tokens\\\":5}\") ;;\n  eagle3) EXTRA=(--speculative-config \"{\\\"method\\\":\\\"eagle3\\\",\\\"model\\\":\\\"$EAGLE\\\",\\\"num_speculative_tokens\\\":3}\") ;;\n  *) echo \"unknown mode: $MODE (baseline|ngram|draft|eagle3)\" >&2; exit 2 ;;\nesac\n\nexec vllm serve \"$TARGET\" \"${COMMON[@]}\" ${EXTRA[@]+\"${EXTRA[@]}\"}\n```\n\nThree details are worth explaining. The JSON goes inside single quotes for the static modes so the shell leaves the double quotes alone, and inside an escaped double-quoted string where a variable must expand. The `${EXTRA[@]+\"${EXTRA[@]}\"}` expansion keeps the baseline mode working on older bash versions, which treat an empty array as unset under `set -u`. And `exec` replaces the shell with the server process, so the process ID you capture later is the server’s own.\n\nThe num_speculative_tokens values (5 for n-gram and draft, 3 for EAGLE3) are starting points, not truths. Step 12 and the advanced tips show how to sweep them.\n\n**Step 7: A streaming client that measures what users feel.** The benchmark sends each prompt to the OpenAI-compatible chat endpoint with temperature 0 and streaming on. It records time to first token, total time, text and the token count. One design choice matters a lot: the client reads the token count from the usage block that the server sends at the end of the stream, because a single streamed chunk can carry several tokens when speculation accepts a run of them. Counting chunks would make speculative runs look slower than they are.\n\n``` python\nimport argparse\nimport asyncio\nimport json\nimport time\nfrom pathlib import Path\n\nimport httpx\n\ndef load_rows(kind, limit):\n    lines = Path(\"data/prompts.jsonl\").read_text().splitlines()\n    rows = [json.loads(line) for line in lines]\n    rows = [r for r in rows if kind == \"all\" or r[\"kind\"] == kind]\n    return rows[:limit]\n\ndef pct(values, p):\n    ordered = sorted(values)\n    return ordered[min(len(ordered) - 1, int(p / 100 * len(ordered)))]\n\nasync def one(client, url, model, row):\n    body = {\n        \"model\": model,\n        \"messages\": row[\"messages\"],\n        \"max_tokens\": row[\"max_tokens\"],\n        \"temperature\": 0,\n        \"stream\": True,\n        \"stream_options\": {\"include_usage\": True},\n    }\n    start = time.perf_counter()\n    first = None\n    pieces = []\n    tokens = 0\n    async with client.stream(\"POST\", url, json=body) as resp:\n        resp.raise_for_status()\n        async for line in resp.aiter_lines():\n            if not line.startswith(\"data: \"):\n                continue\n            payload = line[6:]\n            if payload == \"[DONE]\":\n                break\n            event = json.loads(payload)\n            if event.get(\"usage\"):\n                tokens = event[\"usage\"][\"completion_tokens\"]\n            for choice in event.get(\"choices\", []):\n                piece = choice.get(\"delta\", {}).get(\"content\")\n                if piece:\n                    if first is None:\n                        first = time.perf_counter()\n                    pieces.append(piece)\n    end = time.perf_counter()\n    first = first or end\n    decode_s = max(end - first, 1e-9)\n    return {\n        \"id\": row[\"id\"],\n        \"ttft\": first - start,\n        \"total\": end - start,\n        \"tokens\": tokens,\n        \"decode_tok_s\": max(tokens - 1, 0) / decode_s,\n        \"text\": \"\".join(pieces),\n    }\n\nasync def run(args):\n    rows = load_rows(args.kind, args.limit)\n    url = f\"http://localhost:{args.port}/v1/chat/completions\"\n    gate = asyncio.Semaphore(args.concurrency)\n\n    async with httpx.AsyncClient(timeout=600) as client:\n        await one(client, url, args.model, rows[0])  # warm-up, not recorded\n\n        async def guarded(row):\n            async with gate:\n                return await one(client, url, args.model, row)\n\n        wall_start = time.perf_counter()\n        results = await asyncio.gather(*(guarded(r) for r in rows))\n        wall = time.perf_counter() - wall_start\n\n    total_tokens = sum(r[\"tokens\"] for r in results)\n    summary = {\n        \"label\": args.label,\n        \"kind\": args.kind,\n        \"concurrency\": args.concurrency,\n        \"requests\": len(results),\n        \"agg_tok_s\": total_tokens / wall,\n        \"p50_decode_tok_s\": pct([r[\"decode_tok_s\"] for r in results], 50),\n        \"p50_ttft\": pct([r[\"ttft\"] for r in results], 50),\n        \"p95_ttft\": pct([r[\"ttft\"] for r in results], 95),\n    }\n    Path(\"results\").mkdir(exist_ok=True)\n    out = Path(f\"results/{args.label}_{args.kind}_c{args.concurrency}.json\")\n    out.write_text(json.dumps({\"summary\": summary, \"rows\": results}, indent=1))\n    print(\n        f\"{args.label:<9} {args.kind:<5} c={args.concurrency:<3} \"\n        f\"agg={summary['agg_tok_s']:.1f} tok/s  \"\n        f\"p50 decode={summary['p50_decode_tok_s']:.1f} tok/s  \"\n        f\"p50 ttft={summary['p50_ttft'] * 1000:.0f} ms\"\n    )\n\nif __name__ == \"__main__\":\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--label\", required=True)\n    ap.add_argument(\"--kind\", default=\"all\", choices=[\"all\", \"code\", \"chat\", \"json\"])\n    ap.add_argument(\"--concurrency\", type=int, default=1)\n    ap.add_argument(\"--limit\", type=int, default=32)\n    ap.add_argument(\"--port\", type=int, default=8000)\n    ap.add_argument(\"--model\", default=\"meta-llama/Llama-3.1-8B-Instruct\")\n    asyncio.run(run(ap.parse_args()))\n```\n\nTwo more notes on the client. It sends one unrecorded warm-up request so CUDA graph capture and caches do not pollute the first sample, and it uses a semaphore to hold exactly the concurrency you asked for. The `--limit` default of 32 keeps one run short. Raise it to 100 or more when you want tighter confidence on a final measurement.\n\n## Step 8: Measure the Baseline Twice\n\nStart the plain server in one terminal.\n\n```\ncd ~/specdec-lab && source .venv/bin/activate\n./serve.sh baseline\n```\n\nStartup takes a while the first time because the engine loads weights and captures graphs. Wait until the log says the application has started, or poll the health endpoint from a second terminal.\n\n```\nuntil curl -sf localhost:8000/health > /dev/null; do sleep 3; done && echo healthy\n```\n\nNow run the baseline at concurrency 1 on the code prompts, and run it twice.\n\n```\npython bench.py --label baseline --kind code --concurrency 1\npython bench.py --label baseline --kind code --concurrency 1\n```\n\nThe line format looks like this. The values here are placeholders that show the layout, not measurements.\n\n```\nbaseline  code  c=1   agg=NNN.N tok/s  p50 decode=NNN.N tok/s  p50 ttft=NN ms\nbaseline  code  c=1   agg=NNN.N tok/s  p50 decode=NNN.N tok/s  p50 ttft=NN ms\n```\n\nThe two runs should agree within a few percent. The gap between them is your noise floor, and any speculative speedup smaller than that gap is not real. If the two runs differ by 10 percent or more, something else is using the GPU, the card is power-throttling or the clocks are settling. Fix that first. A laptop GPU and a card in a hot case are the usual culprits.\n\nCheck the server’s metrics endpoint once. In baseline mode it should contain no speculative decoding counters.\n\n```\ncurl -s localhost:8000/metrics | grep -c spec_decode || true\n```\n\nA count of 0 confirms that the baseline is clean. Stop the server with Ctrl+C before moving on, and confirm with `nvidia-smi` that memory has been released.\n\n## Steps 9 to 11: Turn On N-Gram, Draft-Model and EAGLE3 Speculation\n\nEach method follows the same loop: start the server in its mode, wait for health, run the same benchmark command, read the acceptance counters, stop the server. Use the same three prompt kinds so the comparison stays fair.\n\n### Step 9: N-gram lookup\n\n```\n./serve.sh ngram\n# in a second terminal\npython bench.py --label ngram --kind code --concurrency 1\npython bench.py --label ngram --kind chat --concurrency 1\ncurl -s localhost:8000/metrics | grep spec_decode\n```\n\nPredict the result before you read it. Code edits and JSON extraction copy heavily from the prompt, so n-gram lookup should do well there. Fresh chat answers should show a poor acceptance rate, and the speedup there should sit near 1.0x or slightly below. If chat comes out clearly faster, check that your prompts are not secretly repetitive. The metrics grep prints counters for drafted and accepted tokens. The exact metric names change between versions, so grep for spec_decode and divide accepted by drafted yourself. An output shaped like the following is what you want to see.\n\n```\nvllm:spec_decode_num_drafts_total{...} NNNN.0\nvllm:spec_decode_num_draft_tokens_total{...} NNNNN.0\nvllm:spec_decode_num_accepted_tokens_total{...} NNNNN.0\n```\n\n### Step 10: A small draft model\n\n```\n./serve.sh draft\n# in a second terminal\npython bench.py --label draft --kind chat --concurrency 1\npython bench.py --label draft --kind code --concurrency 1\n```\n\nThis mode loads two models, so startup is slower and memory use is higher. If it runs out of memory, shrink `--max-model-len` to 4096 in `serve.sh` or raise `--gpu-memory-utilization` a little, but change the same flag for every mode so the comparison stays honest. A same-family 1B draft typically agrees with its 8B sibling on easy, high-confidence tokens such as punctuation, common words and boilerplate, and disagrees on content choices. Expect a better acceptance rate than n-gram on chat, and a worse one than n-gram on code that copies the prompt verbatim.\n\n### Step 11: EAGLE3\n\n```\n./serve.sh eagle3\n# in a second terminal\npython bench.py --label eagle3 --kind chat --concurrency 1\npython bench.py --label eagle3 --kind code --concurrency 1\n```\n\nEAGLE3 drafts with a head that reads the target’s own hidden states, so its proposals track the target more closely than a separate small model does. That is the reason it often posts the highest acceptance rate on open-ended text. It also depends entirely on the head matching the target and the chat template. If acceptance sits near zero, you probably loaded a head trained for a different model or a different instruction format. The troubleshooting table covers it.\n\n## Step 12: Sweep Concurrency and Verify Lossless Output\n\nRunning each mode by hand works for one data point. A sweep across prompt kinds and concurrency levels 1, 4 and 16 is what exposes the crossover where speculation stops paying. The runner script automates the whole matrix: it starts each server, waits for health, runs nine benchmarks, saves the speculative counters and stops the server.\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\ncd \"$(dirname \"$0\")\"\nmkdir -p logs results data\n[ -f data/prompts.jsonl ] || python make_prompts.py\n\nfor MODE in baseline ngram draft eagle3; do\n  echo \"== starting $MODE\"\n  ./serve.sh \"$MODE\" > \"logs/server_$MODE.log\" 2>&1 &\n  PID=$!\n  until curl -sf localhost:8000/health > /dev/null; do\n    kill -0 \"$PID\" 2> /dev/null || { echo \"server for $MODE died, read logs/server_$MODE.log\"; exit 1; }\n    sleep 5\n  done\n  for KIND in code chat json; do\n    for C in 1 4 16; do\n      python bench.py --label \"$MODE\" --kind \"$KIND\" --concurrency \"$C\"\n    done\n  done\n  curl -s localhost:8000/metrics | grep spec_decode > \"results/metrics_$MODE.txt\" || true\n  kill \"$PID\"\n  wait \"$PID\" 2> /dev/null || true\n  sleep 10\ndone\npython compare.py | tee results/summary.txt\n```\n\nThe comparison script then reads the saved results and prints one row per method, kind and concurrency level, with speedup relative to the baseline at the same settings and a count of requests whose text matches the baseline exactly.\n\n``` python\nimport json\nfrom pathlib import Path\n\nLABELS = [\"baseline\", \"ngram\", \"draft\", \"eagle3\"]\nKINDS = [\"code\", \"chat\", \"json\"]\nCONCURRENCY = [1, 4, 16]\n\ndef load(label, kind, conc):\n    path = Path(f\"results/{label}_{kind}_c{conc}.json\")\n    return json.loads(path.read_text()) if path.exists() else None\n\nprint(f\"{'kind':<6}{'conc':>5}  {'method':<9}{'tok/s':>9}{'speedup':>9}{'p50 ttft ms':>13}{'same text':>11}\")\nfor kind in KINDS:\n    for conc in CONCURRENCY:\n        base = load(\"baseline\", kind, conc)\n        if not base:\n            continue\n        base_rate = base[\"summary\"][\"agg_tok_s\"]\n        for label in LABELS:\n            run = load(label, kind, conc)\n            if not run:\n                continue\n            s = run[\"summary\"]\n            same = sum(1 for a, b in zip(base[\"rows\"], run[\"rows\"]) if a[\"text\"] == b[\"text\"])\n            total = len(base[\"rows\"])\n            print(\n                f\"{kind:<6}{conc:>5}  {label:<9}{s['agg_tok_s']:>9.1f}\"\n                f\"{s['agg_tok_s'] / base_rate:>8.2f}x{s['p50_ttft'] * 1000:>13.0f}{same:>7}/{total}\"\n            )\n```\n\nRun everything with one command.\n\n```\n./run_all.sh\n```\n\nThe output has this shape. Again, the figures are placeholders that show the layout.\n\n```\nkind   conc  method       tok/s  speedup  p50 ttft ms  same text\ncode      1  baseline     NNN.N    1.00x           NN     32/32\ncode      1  ngram        NNN.N    N.NNx           NN     NN/32\ncode      1  draft        NNN.N    N.NNx           NN     NN/32\ncode      1  eagle3       NNN.N    N.NNx           NN     NN/32\ncode      4  baseline     NNN.N    1.00x           NN     32/32\n...\n```\n\n**How to read it.** Start with the identical-text column. If a method matches the baseline on nearly every request, the lossless property is holding in practice. A handful of mismatches at higher concurrency is normal, because batch composition changes floating-point reduction order and can flip a near-tie. If most of the text differs, treat that as a bug in your configuration rather than a feature. Then read the speedup column per concurrency level. Most setups show the largest gains at concurrency 1 and a shrinking or negative gain at 16.\n\n**Turn the table into a capacity decision.** A per-user speedup is not a capacity gain. If a method raises P50 decode speed at concurrency 1 but leaves aggregate tok/s flat at concurrency 16, you bought a better experience for quiet hours and nothing for peak load. Write both numbers down. If your peak concurrency sits in the low single digits, trust the per-user column. If you run dozens of streams per GPU, trust the aggregate column and be ready to find that plain decoding wins.\n\n**Check acceptance before you tune anything else.** When a method underperforms, divide accepted draft tokens by proposed draft tokens using the saved metrics files. An acceptance rate below roughly one half usually means the proposer does not fit the workload, and shortening the draft will not rescue it. A rate above about three quarters with a disappointing speedup points at overhead instead, such as an expensive draft, too little spare GPU capacity or a long context that makes the draft’s own cache heavy.\n\nUse the table below as a decision aid once you have your own numbers. The middle column is what the mechanics predict, so it is a hypothesis you are testing, not a result.\n\n| Workload | Method to try first | What to check in your results | \n|---|---|---|\n| Code rewriting, refactors, edits | N-gram or suffix | Acceptance rate on code prompts and the speedup at concurrency 1 and 4 | \n| Structured extraction to JSON | N-gram | Whether short outputs leave enough decode time to matter at all | \n| Open-ended chat at low concurrency | EAGLE3 or a draft model | Speedup in P50 decode tok/s with TTFT unchanged | \n| Busy API at high concurrency | Often no speculation | Aggregate tok/s at concurrency 16 and above versus the baseline | \n| Mixed traffic | Measure per route | Whether one config helps every kind or hurts one of them | \n\nIf you need to prove to someone else that the model you serve is the model you claim, speculation is a point to disclose. Output equality at temperature 0 is evidence, not proof, and provable-inference research such as [FlashProve](https://shattered.io/anthropic-flashprove-provable-inference-72x-overhead-2026/) targets exactly the gap between a claimed model and the one actually serving tokens. For ordinary serving, the identical-output count is enough.\n\n## Beyond Draft Models: MTP Heads and llama.cpp for Single-User Setups\n\n**MTP in vLLM.** If your target model ships with native multi-token-prediction heads, you do not need a second checkpoint. The vLLM configuration is a single JSON object with the method set to mtp and one speculative token.\n\n```\nvllm serve YOUR_TARGET_WITH_MTP_HEADS \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'\n```\n\nvLLM’s own example uses XiaomiMiMo/MiMo-7B-Base. Google also released MTP drafters for the Gemma 4 family under the Apache 2.0 license according to its announcement, which positions them as usable across Transformers, MLX, vLLM, SGLang, Ollama and Google AI Edge Gallery. To add MTP to the lab, add a fifth case to `serve.sh` and pass the new target through the `TARGET` variable, but remember that the identical-text comparison only makes sense against a baseline of the same target.\n\n**Other engines.** SGLang has its own [speculative decoding documentation](https://docs.sglang.io/docs/advanced_features/speculative_decoding) with a similar set of methods, and NVIDIA’s NIM containers expose speculation too. One trap there: NIM for LLMs 2.1.5 updated its vLLM backend to 0.31.0, so the container’s flags and features trail what you installed from PyPI at 0.33.0. Read the container’s own docs, not the upstream ones.\n\n**llama.cpp for a single GPU or CPU box.** If you run a local assistant on a desktop and care about one user’s speed, llama.cpp is a natural fit. It supports running a main model with a smaller draft model in llama-server. The flags below are the commonly used ones, but they shift between builds, so confirm them against `llama-server --help` on your build. The project reportedly marked v0.9.0 as its latest stable tag on November 4, 2026, with development builds publishing almost daily after that.\n\n```\nllama-server \\\n  -m models/target-Q4_K_M.gguf \\\n  -md models/draft-Q8_0.gguf \\\n  --draft-max 16 --draft-min 1 \\\n  -ngl 99 -ngld 99 -c 8192 --port 8080\n```\n\nThe same measurement discipline applies. Run the same prompts with and without the `-md` flag, compare per-request decode speed and check that the output text is unchanged at temperature 0. Point the `--port` option of the benchmark script at your running llama-server instance.\n\n## Common Pitfalls That Ruin Speculative Decoding Benchmarks\n\n- **Counting stream chunks as tokens.** Speculation delivers several tokens per chunk. Read the usage block, as the client does, or your speculative runs will look slower than the baseline.\n- **Testing at one concurrency only.** A method that wins at concurrency 1 can lose at 16. Report the sweep, not a single flattering row.\n- **Skipping the noise floor.** Run the baseline twice. A 3 percent gain inside a 5 percent wobble is not a result.\n- **Using a draft with a different tokenizer.** The verification step compares token IDs. A mismatched vocabulary either errors out or produces near-zero acceptance. Use the same family.\n- **Setting the speculative token count too high.** More proposed tokens means more wasted work when acceptance is mediocre. Start at 3 to 5 and sweep.\n- **Following a Medusa or MLP speculator guide.** Those paths were reportedly removed in vLLM 0.33.0. Check the date on any guide you follow.\n- **Expecting identical text with sampling on.** Temperature above 0 changes the text run to run by design. Keep temperature at 0 for the equality check.\n- **Letting the draft starve the KV cache.** Draft weights and the draft’s own cache come out of the same VRAM budget. Long contexts plus a draft is where out-of-memory errors appear.\n- **Comparing against a warm server.** Prefix caching can make a second run faster for reasons unrelated to speculation. The runner restarts the server per method, which avoids that.\n- **Trusting a vendor speedup claim for your stack.** A claim like Google’s up to 3x is measured on specific hardware with specific prompts. Reproduce it or discount it.\n\n## Troubleshooting: 12 Problems and Fixes\n\n| Symptom | Likely cause | Fix | \n|---|---|---|\n| 401 or 403 when downloading a Llama checkpoint | Gated repo or missing token | Accept the license on the model page, create a read token and export HF_TOKEN | \n| Import error mentioning a CUDA library right after install | Driver too old for the wheel | Update the NVIDIA driver, or install a vLLM build that matches your CUDA stack. Note that 0.33.0 dropped CUDA 12.9 | \n| Error parsing speculative_config | JSON quoting broken by the shell | Wrap the JSON in single quotes, or escape inner double quotes as the script does | \n| Unknown speculative method | vLLM older than 0.33.0 | Run `vllm --version` , then upgrade to 0.33.0 inside the venv | \n| Vocabulary or tokenizer mismatch error with a draft | Draft from a different model family | Choose a draft that shares the target’s tokenizer | \n| Out-of-memory at startup in draft mode | Draft weights plus cache exceed the budget | Lower `--max-model-len` or use a quantized or smaller draft, and keep the flag identical across modes | \n| Speculative run slower than baseline | Low acceptance or high concurrency | Lower `num_speculative_tokens` , try n-gram on copy-heavy prompts, or test at lower concurrency | \n| Outputs differ from the baseline at temperature 0 | Numeric drift from batching | Count how many differ and where they diverge. A few late divergences are normal, while early divergence in most requests is a config bug | \n| No spec_decode lines in the metrics | Baseline mode, or renamed counters | Confirm the mode, then grep for spec without the suffix | \n| EAGLE3 acceptance near zero | Head trained for a different target or template | Use the head published for your exact target and keep the chat template identical | \n| Server never reports healthy | Still downloading, loading or capturing graphs | Read `logs/server_MODE.log` , wait out the first start and pre-download checkpoints in Step 4 | \n| GPU memory still used after stopping | A child process survived the kill | Run `pkill -f 'vllm serve'` and check`nvidia-smi` before the next mode | \n\n## Advanced Tips for Tuning and Training Your Own Drafters\n\n**Sweep the draft length.** The best number of speculative tokens depends on the workload. Add a loop around `serve.sh` that tries 2, 3, 4, 6 and 8 for each method and plot P50 decode tok/s against it. Copy-heavy workloads usually tolerate longer drafts, while free-form chat peaks earlier.\n\n**Train a drafter on your own traffic.** A generic draft head matches a generic distribution. If your product produces a narrow kind of text, a drafter trained on that text raises acceptance. vLLM’s Speculators project is documented as a library for training draft models that integrate with vLLM, according to the project’s docs. NVIDIA’s NeMo AutoModel documentation, according to its [EAGLE-3 training recipe page](https://docs.nvidia.com/nemo/automodel/latest/nemo-automodel/nemo_automodel/recipes/speculative_decoding/train_eagle3), covers EAGLE-3 training for Llama-style dense models and Qwen3-style mixture-of-experts backbones. On AMD hardware, AMD’s ROCm blog describes [training EAGLE3 draft models on Instinct MI350X](https://rocm.blogs.amd.com/artificial-intelligence/eagle3-training/README.html) with Quark 0.13, dated October 27, 2026.\n\n**[Fine-tune](https://shattered.io/fine-tune-llm-qlora-12-steps-2026/) built-in MTP heads.** NVIDIA’s NeMo documentation reportedly covers fine-tuning a model’s native MTP heads for domain-specific self-speculative decoding. That path keeps a single checkpoint, which simplifies deployment.\n\n**Know which verification modes are lossy.** The 0.33.0 notes reportedly add an opt-in lossy verification policy and refiners for block-diffusion drafters. Those can raise throughput by accepting tokens the target would not have chosen. Keep them off for evals, for anything with compliance requirements and for any run where you want the identical-text count to mean something.\n\n**Switch speculation off under load.** Since the gain shrinks as batches grow, a production router can send latency-sensitive interactive traffic to a speculative deployment and bulk traffic to a plain one. Measure the crossover in your own sweep, then encode it as a routing rule instead of guessing.\n\n**Test with your real output lengths.** Speculation pays off per decoded token, so short answers leave little to save. A JSON extraction that emits 40 tokens spends most of its wall time on prompt processing, and no draft method changes that. Keep the token caps in `make_prompts.py` close to what production sends.\n\n**Watch long contexts.** Verification reads the whole KV cache for every candidate, and a draft model keeps its own cache. If your product runs long documents through the model, rerun the sweep at that context length. A configuration that wins at 2,000 tokens of context can lose at 30,000.\n\n**Export acceptance rate to your dashboards.** A drop in acceptance usually means the prompt mix shifted, a model got swapped or a checkpoint is mismatched. It is a cheap early-warning signal, and the counters already exist on the metrics endpoint.\n\n**Keep the lab around.** Re-run `./run_all.sh` whenever you upgrade vLLM, swap the target or change GPUs. Results from one stack do not transfer to the next. More walkthroughs live in the [AI and machine learning section](https://shattered.io/category/ai-machine-learning/) for the model-side work that feeds a harness like this one.\n\n## The Complete Working Project\n\nThe finished lab is eight small files. Nothing outside this layout is required.\n\n```\nspecdec-lab/\n  requirements.txt     vllm==0.33.0 and httpx\n  make_prompts.py      builds data/prompts.jsonl (112 prompts, 3 kinds)\n  serve.sh             baseline | ngram | draft | eagle3\n  bench.py             streaming client, writes results/*.json\n  compare.py           speedup and identical-text table\n  run_all.sh           runs the full matrix and saves summary.txt\n  data/  results/  logs/\n```\n\nThe end-to-end sequence from an empty machine looks like this.\n\n```\nmkdir -p ~/specdec-lab && cd ~/specdec-lab\nuv venv --python 3.12 .venv && source .venv/bin/activate\nuv pip install -r requirements.txt\nexport HF_TOKEN=hf_your_read_token_here\nhf download meta-llama/Llama-3.1-8B-Instruct\nhf download meta-llama/Llama-3.2-1B-Instruct\nhf download vllm-project/EAGLE3-LLaMA3.1-Instruct-8B\npython make_prompts.py\n./run_all.sh\ncat results/summary.txt\n```\n\nWhen it finishes, `results/summary.txt` holds the comparison and `results/metrics_*.txt` holds the speculative counters for each mode. Keep both next to the vLLM version and driver version you ran, because a speedup number without its stack is not worth saving. Write down the decision in one sentence per workload, for example which method you will deploy for interactive chat and which route stays plain.\n\nA final thought on scope. Speculative decoding is a latency tool for memory-bound serving, and the honest answer for many deployments is that it helps one route and does nothing for another. The lab exists so that answer comes from your GPU and your prompts, not from somebody else’s chart.\n\n## FAQ\n\n**What is speculative decoding in simple terms?** A small, fast proposer guesses the next several tokens and the large model checks all of them in one pass. Accepted guesses are free speed, and rejected ones are discarded so the final text matches what the large model would have written alone.\n\n**Does speculative decoding change the model’s output?** In greedy mode, accepted tokens must equal the target’s own choice, so the text should be the same as without speculation. With sampling, standard rejection-sampling rules preserve the target distribution. Floating-point differences can still flip rare near-ties, which is why the lab counts identical outputs. Opt-in lossy verification modes are the exception and should stay off for evals.\n\n**How much faster will my model get?** It depends on acceptance rate, draft cost, GPU and concurrency. Google claims up to 3x for its Gemma 4 drafters, and the back-of-envelope math in the first section gives a ceiling of roughly 2.4x for an 80 percent acceptance rate with a cheap draft. Your measurement from Step 12 is the only number that counts for your stack.\n\n**Which method should I try first?** For copy-heavy work such as code edits and structured extraction, start with n-gram lookup because it needs no extra model. For open-ended chat at low concurrency, try EAGLE3 if a head exists for your target, or a same-family draft model. If your model ships native MTP heads, try those before downloading anything.\n\n**Why did speculation make my server slower?** Most often the batch is large enough that decode is no longer memory-bound, so verifying extra tokens costs real compute. Low acceptance is the second cause. Lower the draft length, test at lower concurrency and compare acceptance rates before blaming the engine.\n\n**Can I use a draft model from a different family?** Not reliably. The verifier compares token IDs, so the draft needs the same tokenizer as the target. Models from the same family usually qualify. A different family either fails at startup or produces acceptance near zero.\n\n**Does speculative decoding need more GPU memory?** Yes for draft models and EAGLE heads, because their weights and cache live next to the target’s. N-gram lookup adds almost none. Plan for the extra footprint and test with your real context length, since long contexts are where out-of-memory errors show up first.\n\n**Is this worth doing on a local desktop?** For a single-user assistant, usually yes, because single-stream decode is exactly where speculation helps most. Try llama.cpp’s draft-model flags or vLLM’s n-gram mode, run the same prompts both ways and keep whichever gives a better per-request decode speed with unchanged text.", "url": "https://wpnews.pro/news/speculative-decoding-tutorial-12-steps-75-min-2026", "canonical_source": "https://shattered.io/speculative-decoding-tutorial-vllm-12-steps-2026/", "published_at": "2026-11-13 10:25:49+00:00", "updated_at": "2026-11-13 10:52:25.844323+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "ai-tools"], "entities": ["Shattered.io", "Google", "Gemma 4", "EAGLE3", "vLLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/speculative-decoding-tutorial-12-steps-75-min-2026", "markdown": "https://wpnews.pro/news/speculative-decoding-tutorial-12-steps-75-min-2026.md", "text": "https://wpnews.pro/news/speculative-decoding-tutorial-12-steps-75-min-2026.txt", "jsonld": "https://wpnews.pro/news/speculative-decoding-tutorial-12-steps-75-min-2026.jsonld"}}