One of the hottest topics in LLM inference acceleration right now is Speculative Decoding.
DSpark claims 60%–85% single-user speedup at the same throughput. Google has published a stream of research on it — SpecTr, block verification, SpecRouter, and more.
Sounds great, right? A small model (draft model) writes a draft, the large model batch-verifies it, and speed goes up.
But if you're a production engineer looking at this, two questions immediately pop up:
"Block generation — doesn't that amplify hallucinations?"
"You're running an extra model regardless of hit or miss — isn't that wasted compute?"
These two questions hit right at the core of Speculative Decoding's math promise and its engineering cost.
Let's run the numbers — no hype, no FUD.
This is the most misunderstood part of Speculative Decoding. Intuitively: "guess 5 tokens, one wrong and the rest are junk" — correct. But Speculative Decoding is designed precisely to prevent "junk" from becoming "wrong."
The verification mechanism is token-by-token, not "accept all or reject all."
The draft model generates a candidate block: [t1, t2, t3, t4, t5]
. The target model verifies all 5 positions in one forward pass. The result:
Every output token has been confirmed by the target model. No hallucination is "amplified" — it's simply truncated at the first error. In terms of probability distribution, Speculative Decoding's output is mathematically equivalent to the target model's autoregressive output — a provable property.
So the answer to question one is: lossless quality. The promise holds.
One caveat: this equivalence assumes the draft and target models share the
same tokenizer. If they differ (e.g., one uses BPE, the other Unigram), the verification process will have alignment overhead. It's not a bug in Speculative Decoding, but something to verify before deploying to production.
The second question is harder to answer.
"You're running an extra model regardless" — how do we account for that cost?
First, a premise: a small model's forward pass typically costs 1/10 to 1/20 of the target model's. That's because the core assumption of Speculative Decoding is that the draft model is small — a common pairing is a 7B drafting for a 70B. All the math below builds on this assumption.
Let's walk through three scenarios with a draft length of 5:
Scenario A: Full hit (best case)
| Without SD | With SD | |
|---|---|---|
| Target model runs | 5 | 1 |
| Draft model runs | 0 | 1 |
| Net | 5 target runs | 1 target + 1 draft |
Saving: 4 target runs minus 1 draft run.
**Scenario B: Full miss (worst case)**
| Without SD | With SD | |
|---|---|---|
| Target model runs | 5 | 1 (verification) + 5 (regeneration) |
| Draft model runs | 0 | 1 | | Net | 5 target runs | 6 target + 1 draft |
Result: slower than autoregressive, with a wasted draft run on top.
Scenario C: Partial hit (common case) | Without SD | With SD | |
|---|---|---|
| Target model runs | 5 | 1 (verification) + (5 - hits) (regeneration) |
| Draft model runs | 0 | 1 |
| Net | 5 target runs | (6 - hits) target + 1 draft |
Net benefit: positive only when `hits > 1 + (draft_cost / target_cost)`
.
See the pattern? Speculative Decoding isn't "always faster." It's a high-risk, high-reward bet. Win and you save compute. Lose and you pay extra.
Let's formalize the math above into a single inequality.
Let:
k
= draft length (how many tokens per guess)α
= compute ratio of draft model to target model (for a 7B/70B pair, α ≈ 0.05–0.1
)β
= verification phase overhead per tokena
= average acceptance length (how many tokens pass verification per round)Speculative Decoding is strictly better than autoregressive when:
a > 1 + α + β
Or in words: the average acceptance length must exceed 1 (at least one token accepted per round), and the surplus must cover the draft model and verification overhead.
a = 5
(all hit) → `a = 1`
(one hit) → `a < 1`
(zero hits) → How to pick k? Too small and the speedup is negligible. Too large and you waste compute on tail tokens that are almost certainly rejected. Engineering experience:
The distribution shift trap. If the task distribution is far from the draft model's training distribution — say, using a 7B to draft poetry for a 70B — the 7B has no idea how the 70B will choose its words. Acceptance rate can drop below 10%. At that point a < 1
, and Speculative Decoding is strictly worse than autoregressive — and it gets worse as k
increases. This is the single most important thing to watch for in production.
All Speculative Decoding does is play this inequality game, round after round.
A quick reality anchor: in practice, well-matched draft/target pairs (same family, similar training data) achieve a = 2.5–4.0
on code and structured text tasks — comfortably above the 1 + α + β
threshold. Unmatched pairs (different model families, different tokenizers, or high-entropy tasks like free-form dialogue) often land at a = 1.0–1.5
, right in the marginal zone where overhead eats the gain. This is why your mileage varies more by task than by model size.
Before you trust any vendor's benchmark, measure your own a
.
Here's what you do:
Step 1: Instrument the verification boundary. Insert a logging hook between the draft model and the target model's verification pass. For each request, log the draft length k
, the acceptance length a
, and the number of regeneration steps. Any inference framework that supports SD (TensorRT-LLM, vLLM with speculative decoding, HF generate()
with assistant_model
) exposes these counters — or you can patch them in ~50 lines.
Step 2: Collect 500+ samples per task type. Don't average across all traffic — your code completion requests and your creative writing requests will have drastically different a
values. Split by: task category, prompt length bucket, response length bucket. 500 samples per bucket gives you a stable mean and a useful p50/p90/p99 spread.
Step 3: Check the worst decile. The mean a
might be 3.2, but if the bottom 10% of requests have a < 1
, those requests are paying more than they would without SD. In a latency-sensitive system, the p10 a
matters more than the mean.
Step 4: Run the inequality per bucket. Plug each bucket's a
into a > 1 + α + β
. If code completion passes but free-form dialogue fails, you have a deployment strategy: enable SD for the code route, disable it for the chat route.
This isn't optional calibration. It's the difference between "SD saves us 40% latency" and "SD makes our p99 worse and we can't figure out why."
Once you understand the inequality above, DSpark's core contribution becomes obvious: Confidence-based Scheduling.
DSpark adds a confidence head to the draft model. For each draft token, it outputs a "survival probability." The scheduler uses this to dynamically decide how many tokens to verify:
In the inequality framework: DSpark dynamically adjusts k via the confidence head — maximizing the expected acceptance length a while minimizing the wasted α overhead.
Win, you accelerate. Lose, you stop the bleeding early. It turns Speculative Decoding from blind betting into informed gambling.
It's not a yes/no question. It's a "depends."
Use it when:
Don't use it when:
A pragmatic rule:
If you're running high-volume LLM inference, Speculative Decoding is worth evaluating. But don't trust the "85% speedup" number. A/B test on your data and your model pair. Measure your actual acceptance rate. Plug it into a > 1 + α + β
.
If it holds, use it. If it doesn't, don't. Simple as that. Speculative Decoding is an elegant mathematical scheme: lossless quality, faster inference, via a draft-verify mechanism.
But lossless ≠ free.
It doesn't amplify hallucinations. But it does add compute overhead. When the hit rate is high, that overhead buys significant acceleration. When the hit rate is low, it doesn't just fail to accelerate — it slows the system down.
The best optimization technique isn't the one that always wins — it's the one you know when to turn off.
Next time you see a Speculative Decoding paper that only reports "X% speedup" without mentioning the acceptance rate or the worst-case behavior — send them this post.