Adaptive speculative decoding on a $300 GPU

Speculative decoding, measured honestly, on a €300 GPU.

Illustrative run; all performance claims come from the bracketed batteries.

Generating one token means re-reading every weight. On an RTX 5060 (448 GB/s) running Qwen3-8B-Q4_K_M (5.03 GB), the ceiling is 448 ÷ 5.03 ≈ 89 tok/s — and the plain baseline already hits 80.1 tok/s (tg128), i.e. 90 % of it, while using less than 1 % of the GPU's compute. Prompt processing runs at 3031 tok/s: a 37.8× gap between what the silicon can chew and what decoding feeds it.

The whole project is one idea: produce several tokens per read of the weights.

All numbers: llama.cpp b10034

(505b1ed1

), CUDA 13.1 (sm_120), -c 4096

, reasoning disabled, temp 0, seed 42, 3 reps (median), power cap pinned at 145 W, GPU monitored at 1 Hz. Every comparison is bracketed (baseline before + after, ratios only — thermal drift is real: we measured −11 % over a session on Windows and a −17 % window on Linux; the brackets kept every ratio honest, max accepted drift 2.6 %).

Generation (ratio vs bracketed baseline, ~76–78 tok/s):

Code/JSON editing (output ≈ copy of the prompt):

Agent loop — 5 successive edits of the same file on a warm server, stop-at-fence: ngram-mod finishes in 14.6 s vs 57.3 s for the baseline (3.92× wall-clock), sustaining 285 tok/s with peaks at 887 tok/s — ~10× above the naive decoding ceiling. Both modes emitted the exact same 4169 tokens: speculation never changes the output (greedy decoding).

The draft model is a bet that pays on structured content. A 0.6B Q8_0 draft nearly doubles math and JSON throughput. The 0.6B alone runs at 407 tok/s vs ~700 theoretical: it iskernel-launch-bound, not bandwidth-bound (which is also why a Q4 draft loses to Q8 — smaller weights don't help when launches dominate).** The prose regression died with the platform.On Windows the draft cost ~2.6 ms/token even when silent (−33 % on prose). On Linux + b10034 the worst case across every content type is 0.99× — the bet became almost free.The editing goldmine is real but fragile. The speedup tracks the copied fraction (1.86× on a heavy rewrite → 9.27× on a near-pure copy) — and it silently vanishes if the model "thinks" first: chain-of-thought is prose, not copy. One run with Qwen3's default thinking enabled cut edits from 5.3× to 1.03×. Capture your outputs; an anomalous acceptance rate is a measurement, not a victory.ngram-simple is the safe floor: no material regression in our benchmark suite**(≥ 0.99× on every content type measured), no second model, and up to 9.27× on edits for free. The stronger*"never below baseline"*phrasing is reserved for therouter invariant, checked byreproduce.sh

(verdictPASS) — not a guarantee about every conceivable prompt.** What didn't work:**EAGLE-3 (AngelSlim head for Qwen3-8B) collapsed on a Q4 target (58 % acceptance on code → 14.8 % on French prose), beaten by the plain 0.6B draft everywhere. We eliminated quantization, conversion, d2t coverage and chat format as causes; the model card itself only claims ~1.7×. Our expectations were wrong, not the head.

A ~250-line client-side dispatcher over llama-server. It doesn't predict which mode wins — it measures: a 48-token probe on the aggressive config, a prompt-hash cache, epoch batching (one server swap per batch), a static EDIT class (big code/JSON block + edit verb → ngram-mod, no probe), ngram-simple as the safety floor, and an EMA collapse detector.

Invariants (checked by reproduce.sh

): ≥ baseline on all 5 content types, and ≥ 90 % of the static champion on code/math/JSON. Current verdict: PASS (5/5, and the router is the champion on code, math and JSON). Live routing: prose→safe (78.6 tok/s), math→aggressive (119.2 tok/s), JSON edit→edit (327.1 tok/s).

Cold-probe caveat. The router classifies a novel prompt with a short 48-token probe on the aggressive config. On that short window a draft model has not yet amortized its speculation (and right after a server swap GPU clocks are still ramping), so the probe can measure the aggressive config below the plain floor and conservatively route to safe

— e.g. plain code, where the draft wins 1.56× over a full generation but loses on 48 tokens. This never costs throughput: safe

(ngram-simple) is ≥ baseline on every content type in our suite, the floor doing its job. demo.py

warms the aggressive server before routing, matching the battery protocol; full-generation numbers remain the source of every performance claim.

./reproduce.sh        # SHA-checks models, builds llama.cpp (CUDA, sm_120), runs the
.venv/bin/python demo.py   # live demo: prose → code → JSON → edit, tok/s counter

Hardware: Ryzen 5 5500, RTX 5060 8 GB (145 W cap), 16 GB RAM, Ubuntu 26.04. Models: Qwen3-8B-Q4_K_M (SHA256 d98cdcbd…5745785

) + Qwen3-0.6B-Q8_0, from Qwen/Qwen3-*-GGUF

. llama.cpp pinned at b10034

(505b1ed15ca8…

).

Note for Ubuntu 26.04: CUDA 13.1's crt/math_functions.h

clashes with glibc's C23 rsqrt

/rsqrtf

declarations; reproduce.sh

documents the two-line header fix.

Bracket everything. Thermal drift (−11 %/session) will otherwise manufacture or hide your effect. Compare ratios, never absolutes.Log watts. Speculative modes draw ~15 W less than baseline at the same cap — efficiency becomes speed whenever there is thermal headroom.Watch VRAM. Squatted VRAM means silent spill: half the speed, zero errors (Windows/WDDM). Linux with-ngl 99 -c 4096

showed no spill; the harness still refuses to run below 6.8 GB free.Capture outputs. An exploding acceptance rate can measure degeneration (the model looping) — or your speedup can silently die to chain-of-thought. Read what the model actually wrote.

Adaptive and edit-oriented speculative decoding are active research areas. FRONDE's niche is deliberately narrow: training-free, lossless (greedy), a single consumer GPU, routing by measurement rather than prediction, and one-command reproduction.

SpecRouter(arXiv:2505.07680) frames inference as adaptive routing acrossmulti-levelchains of draft/verifier models, using real-time profiling and predictive similarity to choose a path. FRONDE routes amongtraining-free modes(one small off-the-shelf draft, or prompt-lookup n-gram) on a single 8 GB GPU, and itmeasuresa short probe instead of predicting.MetaSD(arXiv:2604.05417) selects among multiple trained drafters with alignment feedback, framed as a bandit problem. FRONDE uses no trained drafter beyond a stock 0.6B and requires no additional training.EfficientEdit(arXiv:2506.02780, ASE 2025) targets code editing with anedit-oriented draft modelplus dynamic verification, reporting up to10.4×–13.1×. Our edit speedups (up to ~9× on a near-pure copy) come fromtraining-freeprompt-copy n-gram speculation with no extra model — a simpler mechanism, and we cite their higher numbers to keep ours in honest context.

Developed with heavy assistance from Claude Code; experimental design, hardware runs and release decisions by the maintainer.

MIT (this repository's code — see LICENSE

). Third-party components: THIRD_PARTY_NOTICES.md

. Model weights (Qwen3, Apache-2.0): MODEL_LICENSES.md

.