{"slug": "what-i-gave-up-to-make-speculative-decoding-faster", "title": "What I Gave Up to Make Speculative Decoding Faster", "summary": "A developer implemented mentored decoding in mini-vllm-rs, a Rust inference engine for macOS, to test whether relaxing lossless speculative sampling speeds up generation. Using quantized Qwen2.5 7B as the target and 0.5B as the draft on an Apple M2, the developer found that at a TV budget of 1 — where every draft token is accepted — fixed-4 generation still took 24.4 seconds versus 21.0 seconds for target-only, because draft generation, target verification and cache updates remained. The work maps where speculation beats target-only latency and what distributional fidelity must be sacrificed to get there.", "body_md": "On a creative-writing prompt, my target LLM rejected most tokens proposed by the smaller draft model. Speculative decoding was slower than running the target alone. I wanted to recover that wasted work. How much faster could generation become if I let the draft have more influence over the output?\n\nLossless speculative sampling preserves the target model’s output distribution. Relaxing that requirement might improve acceptance, but the resulting text would need a closer look.\n\nI explored this tradeoff with mentored decoding in [mini-vllm-rs](https://github.com/lun0522/mini-vllm-rs), my Rust inference engine for macOS. Using quantized Qwen2.5 7B as the target and 0.5B as the draft on an Apple M2, I eventually found a faster configuration. Deciding when I would use it turned out to be a different question.\n\nIn an [earlier benchmark](https://github.com/lun0522/mini-vllm-rs/blob/main/benchmarks/speculative_decoding.md), a code-refactoring prompt worked well under greedy decoding. A policy that adaptively adjusted the draft token count usually selected 8-token proposals, achieved 98.5% acceptance, and raised decode throughput by 38.3% over target-only. However, on the creative-writing prompt, the same policy delivered 36.8% lower decode throughput than target-only, also under greedy decoding.\n\nI then explored the creative-writing prompt with multinomial sampling at temperature 0.7 ([sampling benchmark](https://github.com/lun0522/mini-vllm-rs/blob/main/benchmarks/speculative_sampling_tradeoff.md)). Lossless speculative sampling with a fixed draft token count of 4 achieved only 28.6% acceptance.\n\n[Mentored decoding](https://arxiv.org/abs/2609.30474) allows a bounded shift from the target distribution toward the draft. I implemented its linear-interpolation construction.\n\nLet p be the target distribution and q the draft distribution, both after temperature and repetition penalties. Their total-variation distance is:\n\nGiven a TV budget D , the interpolation weight is:\n\nIf d=0 , the distributions match exactly, and we take θ=1 . The resulting output distribution is:\n\nA draft token x sampled from q is accepted with probability:\n\nAfter a rejection, the sampler draws a replacement from the normalized positive residual max(p−q,0) . Together, acceptance and correction produce the distribution π .\n\nAt TV budget 0, this preserves the target distribution. At TV budget 1, the budget covers every possible TV distance, so θ=1 and π=q . Every draft token is then accepted.\n\nIntermediate budgets limit the per-step distributional deviation from the target. They do not measure how much semantic quality the output loses.\n\nI first set the TV budget to 1, so every draft token was accepted. Even then, fixed-4 generation took 24.425 seconds, versus 20.971 seconds for target-only.\n\nRejection was gone, but execution cost remained. Each round still generated 4 draft tokens autoregressively, verified their positions with the target, sampled, and updated both models’ cache state. My implementation emits no extra target bonus token after a fully accepted proposal, so a successful round produced only 4 output tokens.\n\nThe draft-only control took 3.372 seconds. At TV=1, target verification was an expensive way to follow the draft distribution. That endpoint helped diagnose the remaining cost, but I wanted a speedup while retaining more of the target distribution.\n\nTo find where speculation could beat target-only E2E latency, I tested fixed draft token counts of 6 and 8 at full acceptance. I then lowered the TV budget to find the smallest tested value that still produced a clear speedup ([benchmark report](https://github.com/lun0522/mini-vllm-rs/blob/main/benchmarks/speculative_sampling_proposal_breakeven.md)).\n\nThe prompt and temperature stayed fixed. Every request generated 256 tokens with EOS ignored. After warm-up, each setting had 3 independent untraced timing invocations, each covering 3 random seeds.\n\n| Configuration | Acceptance | Avg. rounds/request | Median E2E latency | \n|---|---|---|---|\n| Target-only | — | — | 20.971 s | \n| Fixed-4, TV=1 | 100% | 64 | 24.425 s | \n| Fixed-6, TV=1 | 100% | 43 | 20.587 s | \n| Fixed-8, TV=1 | 100% | 32 | 18.508 s | \n| Fixed-8, TV=0.75 | 93.4% | 34 | 18.820 s | \n| Fixed-8, TV=0.5 | 63.5% | 46 | 22.900 s | \n\nFixed-6 was approximately equal to target-only within observed variability. Fixed-8 clearly crossed the baseline. At full acceptance, it needed 32 verification rounds per request instead of fixed-4’s 64, allowing more output to share each verification call.\n\nVerification also crossed a kernel-dispatch boundary, as in my [continuous-batching experiments](https://dev.to/lun0522/why-continuous-batching-made-my-llm-inference-engine-slower-3pjb). Here, 4-row verification used row-wise GEMV, while full 6- and 8-row proposals used regular quantized matmul. Proposal length and kernel dispatch changed together, so these measurements do not isolate their individual contributions.\n\nFor fixed-8, increasing the budget from 0.5 to 0.75 reduced average verification rounds from 46 to 34. Generation moved from 9.2% slower than target-only to 10.3% sooner. Each seed’s median improved by roughly 8–10% at TV=0.75. To judge what that speedup bought me, I inspected the creative-writing output.\n\nThe prompt places Mara, a mechanic, and Ilyan, an archivist, in a lunar greenhouse abandoned for 20 years to investigate a distress signal. It specifies fresh condensation despite the lack of power, and patches of blue light reflected from Earth.\n\nIn one target-only sample, the characters treat the condensation as evidence that someone may have visited or something is changing. In the TV=0.75 output using the same seed, the roof reveals:\n\n“a thin sliver of blue light streaming through—Earth’s infrared, indicating that it was not the end of this system but only an interruption in its normal operation.”\n\nInfrared is not visible blue light. The passage also jumps from that light to a diagnosis of the system without explaining the connection.\n\nAt TV=1, the same seed produced an identity error. Immediately after Mara explained the greenhouse, the text continued:\n\n“I'm Ilyan Sato, the archivist from the station,” she added.\n\nMara had taken the other character’s name and role.\n\nThe [complete prompt and samples](https://github.com/lun0522/mini-vllm-rs/blob/main/benchmarks/speculative_sampling_proposal_breakeven.md#appendix-complete-prompt-and-output-samples) show different adherence or continuity problems across all 3 TV=0.75 outputs. These are qualitative observations from 1 prompt, and the 256-token cutoff limits judgments about plot development.\n\nI started by trying to recover rejected draft work. Raising acceptance helped, but even 100% acceptance was insufficient with 4-token proposals. Longer proposals finally crossed the latency baseline, and reducing the TV budget showed that complete acceptance was unnecessary.\n\nThe faster setting also changed the output in ways that the timing table could not capture. The TV budget controlled a mathematical distance, while the text exposed physical inconsistencies and broken reasoning. What I gave up was the guarantee of preserving the target distribution. Whether the time saved justified that change required reading the result, not just measuring how quickly it arrived.", "url": "https://wpnews.pro/news/what-i-gave-up-to-make-speculative-decoding-faster", "canonical_source": "https://dev.to/lun0522/what-i-gave-up-to-make-speculative-decoding-faster-1h87", "published_at": "2026-10-28 00:29:36+00:00", "updated_at": "2026-10-28 00:41:16.462043+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "ai-tools"], "entities": ["mini-vllm-rs", "Qwen2.5 7B", "Qwen2.5 0.5B", "Apple M2", "Rust"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-i-gave-up-to-make-speculative-decoding-faster", "markdown": "https://wpnews.pro/news/what-i-gave-up-to-make-speculative-decoding-faster.md", "text": "https://wpnews.pro/news/what-i-gave-up-to-make-speculative-decoding-faster.txt", "jsonld": "https://wpnews.pro/news/what-i-gave-up-to-make-speculative-decoding-faster.jsonld"}}