{"slug": "adaptive-speculative-decoding-on-a-300-gpu", "title": "Adaptive speculative decoding on a $300 GPU", "summary": "A developer achieved up to 9.27× speedup on code editing tasks using adaptive speculative decoding on a €300 RTX 5060 GPU, with a 0.6B draft model nearly doubling math and JSON throughput. The project's ngram-simple method showed no material regression across all content types, while the draft model's prose regression was eliminated on Linux. The approach uses a 250-line client-side dispatcher that measures performance via a 48-token probe rather than predicting the optimal mode.", "body_md": "*Speculative decoding, measured honestly, on a €300 GPU.*\n\n*Illustrative run; all performance claims come from the bracketed batteries.*\n\nGenerating one token means re-reading every weight. On an RTX 5060 (448 GB/s) running\nQwen3-8B-Q4_K_M (5.03 GB), the ceiling is 448 ÷ 5.03 ≈ **89 tok/s** — and the plain\nbaseline already hits **80.1 tok/s** (tg128), i.e. 90 % of it, while using **less than\n1 % of the GPU's compute**. Prompt processing runs at 3031 tok/s: a **37.8× gap**\nbetween what the silicon can chew and what decoding feeds it.\n\nThe whole project is one idea: **produce several tokens per read of the weights.**\n\nAll numbers: llama.cpp `b10034`\n\n(`505b1ed1`\n\n), CUDA 13.1 (sm_120), `-c 4096`\n\n,\nreasoning disabled, temp 0, seed 42, 3 reps (median), power cap pinned at 145 W, GPU\nmonitored at 1 Hz. Every comparison is **bracketed** (baseline before + after, ratios\nonly — thermal drift is real: we measured −11 % over a session on Windows and a −17 %\nwindow on Linux; the brackets kept every ratio honest, max accepted drift 2.6 %).\n\n**Generation** (ratio vs bracketed baseline, ~76–78 tok/s):\n\n| Content | Draft 0.6B (n_max 12, p_min 0.45) | ngram-simple | ngram-mod |\n|---|---|---|---|\n| Python code | 1.56× (122 tok/s) |\n1.01× | 1.00× |\n| French prose | 1.02× | 1.00× | 1.00× |\n| English prose | 0.99× | 1.00× | 1.01× |\n| Math (step-by-step) | 1.97× (153 tok/s) |\n0.99× | 0.98× |\n| JSON | 1.92× (150 tok/s) |\n1.03× | 1.03× |\n\n**Code/JSON editing** (output ≈ copy of the prompt):\n\n| Task | Draft | ngram-simple | ngram-mod (n_match 12) |\n|---|---|---|---|\n| Rename + type hints (heavy rewrite) | 2.76× |\n1.87× | 1.86× |\n| Off-by-one bugfix (near-pure copy) | 2.80× | 9.27× | 9.16× (697 tok/s) |\n| JSON edit (3 changes) | 2.62× | 5.24× | 5.26× (404 tok/s) |\n\n**Agent loop** — 5 successive edits of the same file on a warm server, stop-at-fence:\nngram-mod finishes in **14.6 s vs 57.3 s** for the baseline (**3.92× wall-clock**),\nsustaining **285 tok/s** with peaks at 887 tok/s — ~10× above the naive decoding\nceiling. Both modes emitted the exact same 4169 tokens: speculation never changes\nthe output (greedy decoding).\n\n**The draft model is a bet that pays on structured content.** A 0.6B Q8_0 draft nearly doubles math and JSON throughput. The 0.6B alone runs at 407 tok/s vs ~700 theoretical: it is**kernel-launch-bound**, not bandwidth-bound (which is also why a Q4 draft loses to Q8 — smaller weights don't help when launches dominate).** The prose regression died with the platform.**On Windows the draft cost ~2.6 ms/token even when silent (−33 % on prose). On Linux + b10034 the worst case across every content type is 0.99× — the bet became almost free.**The editing goldmine is real but fragile.** The speedup tracks the copied fraction (1.86× on a heavy rewrite → 9.27× on a near-pure copy) — and it silently vanishes if the model \"thinks\" first: chain-of-thought is prose, not copy. One run with Qwen3's default thinking enabled cut edits from 5.3× to 1.03×. Capture your outputs; an anomalous acceptance rate is a measurement, not a victory.**ngram-simple is the safe floor**:** no material regression in our benchmark suite**(≥ 0.99× on every content type measured), no second model, and up to 9.27× on edits for free. The stronger*\"never below baseline\"*phrasing is reserved for the**router invariant**, checked by`reproduce.sh`\n\n(verdict**PASS**) — not a guarantee about every conceivable prompt.** What didn't work:**EAGLE-3 (AngelSlim head for Qwen3-8B) collapsed on a Q4 target (58 % acceptance on code → 14.8 % on French prose), beaten by the plain 0.6B draft everywhere. We eliminated quantization, conversion, d2t coverage and chat format as causes; the model card itself only claims ~1.7×. Our expectations were wrong, not the head.\n\nA ~250-line client-side dispatcher over llama-server. It doesn't predict which mode\nwins — it **measures**: a 48-token probe on the aggressive config, a prompt-hash\ncache, epoch batching (one server swap per batch), a static EDIT class (big code/JSON\nblock + edit verb → ngram-mod, no probe), ngram-simple as the safety floor, and an\nEMA collapse detector.\n\n**Invariants** (checked by `reproduce.sh`\n\n): ≥ baseline on all 5 content types, and\n≥ 90 % of the static champion on code/math/JSON. Current verdict: **PASS** (5/5, and\nthe router *is* the champion on code, math and JSON). Live routing: prose→safe\n(78.6 tok/s), math→aggressive (119.2 tok/s), JSON edit→edit (327.1 tok/s).\n\n**Cold-probe caveat.** The router classifies a novel prompt with a short 48-token\nprobe on the aggressive config. On that short window a draft model has not yet\namortized its speculation (and right after a server swap GPU clocks are still\nramping), so the probe can measure the aggressive config *below* the plain floor\nand conservatively route to `safe`\n\n— e.g. plain code, where the draft wins 1.56×\nover a full generation but loses on 48 tokens. This never costs throughput: `safe`\n\n(ngram-simple) is ≥ baseline on every content type in our suite, the floor doing its\njob. `demo.py`\n\nwarms\nthe aggressive server before routing, matching the battery protocol; full-generation\nnumbers remain the source of every performance claim.\n\n```\n./reproduce.sh        # SHA-checks models, builds llama.cpp (CUDA, sm_120), runs the\n                      # bracketed benchmark battery, prints the invariant verdict\n.venv/bin/python demo.py   # live demo: prose → code → JSON → edit, tok/s counter\n```\n\nHardware: Ryzen 5 5500, RTX 5060 8 GB (145 W cap), 16 GB RAM, Ubuntu 26.04.\nModels: Qwen3-8B-Q4_K_M (SHA256 `d98cdcbd…5745785`\n\n) + Qwen3-0.6B-Q8_0, from\n`Qwen/Qwen3-*-GGUF`\n\n. llama.cpp pinned at `b10034`\n\n(`505b1ed15ca8…`\n\n).\n\nNote for Ubuntu 26.04: CUDA 13.1's `crt/math_functions.h`\n\nclashes with glibc's C23\n`rsqrt`\n\n/`rsqrtf`\n\ndeclarations; `reproduce.sh`\n\ndocuments the two-line header fix.\n\n**Bracket everything.** Thermal drift (−11 %/session) will otherwise manufacture or hide your effect. Compare ratios, never absolutes.**Log watts.** Speculative modes draw ~15 W less than baseline at the same cap — efficiency becomes speed whenever there is thermal headroom.**Watch VRAM.** Squatted VRAM means silent spill: half the speed, zero errors (Windows/WDDM). Linux with`-ngl 99 -c 4096`\n\nshowed no spill; the harness still refuses to run below 6.8 GB free.**Capture outputs.** An exploding acceptance rate can measure degeneration (the model looping) — or your speedup can silently die to chain-of-thought. Read what the model actually wrote.\n\nAdaptive and edit-oriented speculative decoding are active research areas. FRONDE's\nniche is deliberately narrow: **training-free, lossless (greedy), a single consumer\nGPU, routing by measurement rather than prediction, and one-command reproduction.**\n\n**SpecRouter**([arXiv:2505.07680](https://arxiv.org/abs/2505.07680)) frames inference as adaptive routing across*multi-level*chains of draft/verifier models, using real-time profiling and predictive similarity to choose a path. FRONDE routes among*training-free modes*(one small off-the-shelf draft, or prompt-lookup n-gram) on a single 8 GB GPU, and it*measures*a short probe instead of predicting.**MetaSD**([arXiv:2604.05417](https://arxiv.org/abs/2604.05417)) selects among multiple trained drafters with alignment feedback, framed as a bandit problem. FRONDE uses no trained drafter beyond a stock 0.6B and requires no additional training.**EfficientEdit**([arXiv:2506.02780](https://arxiv.org/abs/2506.02780), ASE 2025) targets code editing with an*edit-oriented draft model*plus dynamic verification, reporting up to**10.4×–13.1×**. Our edit speedups (up to ~9× on a near-pure copy) come from*training-free*prompt-copy n-gram speculation with no extra model — a simpler mechanism, and we cite their higher numbers to keep ours in honest context.\n\n*Developed with heavy assistance from Claude Code; experimental design, hardware runs\nand release decisions by the maintainer.*\n\nMIT (this repository's code — see `LICENSE`\n\n). Third-party components: `THIRD_PARTY_NOTICES.md`\n\n.\nModel weights (Qwen3, Apache-2.0): `MODEL_LICENSES.md`\n\n.", "url": "https://wpnews.pro/news/adaptive-speculative-decoding-on-a-300-gpu", "canonical_source": "https://github.com/Gogo27Gallet/Fronde", "published_at": "2026-07-28 13:01:43+00:00", "updated_at": "2026-07-28 13:22:40.220653+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["RTX 5060", "Qwen3-8B", "llama.cpp", "CUDA", "EAGLE-3", "AngelSlim", "Qwen3"], "alternates": {"html": "https://wpnews.pro/news/adaptive-speculative-decoding-on-a-300-gpu", "markdown": "https://wpnews.pro/news/adaptive-speculative-decoding-on-a-300-gpu.md", "text": "https://wpnews.pro/news/adaptive-speculative-decoding-on-a-300-gpu.txt", "jsonld": "https://wpnews.pro/news/adaptive-speculative-decoding-on-a-300-gpu.jsonld"}}