# Osprey boosts speculative decoding acceptance rates by 16% to 22%

> Source: <https://promptcube3.com/en/threads/9176/>
> Published: 2026-09-10 21:41:53+00:00

# Osprey boosts speculative decoding acceptance rates by 16% to 22%

Speculative decoding usually fails the moment you shift your workload because drafters are too specialized to a single target model. If you train a drafter on a narrow dataset for one LLM and then switch to a different domain or a different target model, the acceptance rate collapses. Osprey fixes this by using a target-agnostic pretraining approach, essentially turning a small pretrained LM into a reusable backbone that only needs a lightweight adaptation step to work with a specific target model.

## How Osprey handles the latency-accuracy trade-off

The main problem with using a pretrained small LM as a drafter is that they are usually too deep to be "fast" enough for speculative decoding. If the drafter takes too long, the overhead kills the speedup. Osprey solves this by pruning the small LM into a shallow backbone and then restoring its language capabilities through next-token pretraining that doesn't depend on a specific target model.

To make this work across different models, they use a few specific technical tricks during the adaptation phase:

- Vocabulary alignment to match the target model.
- Zero-initialized QKV expansion.
- Distillation from the target model's output distribution.

## Real world performance numbers

The most impressive part is that a single pretrained Osprey backbone transfers across completely different model families. Based on the data from arXiv:2609.09338v1, the mean acceptance length increases significantly when moving to out-of-domain or multilingual data.

- **Qwen3-8B:** 16.1% increase in mean acceptance length.
- **Llama-3.3-70B-Instruct:** 21.2% increase in mean acceptance length.
- **MiniMax-M2.5 (229B):** 22.7% increase in mean acceptance length and a 17.5% jump in tokens per second.

## Implementing Osprey in your pipeline

If you are trying to implement this, you can't just plug in a random small model. You have to follow the pruning and restoration sequence first. The workflow looks like this:

1. Start with an off-the-shelf small LM.

2. Prune it to a shallow depth to meet latency requirements.

3. Run target-agnostic next-token pretraining to recover the lost knowledge.

4. Perform the adaptation step (Vocab alignment → QKV expansion → Distillation) for your specific target LLM.

The code for this is hosted on GitHub. You can find the implementation here:

```
https://github.com/LeanModels/Osprey
```

This approach is a massive shift from the "train a new drafter for every model" headache. Instead of spending GPU hours retraining a drafter every time you update your target model or change your prompt distribution, you just run the adaptation step. For those of us running massive models like the 229B MiniMax, that 17.5% increase in tokens per second is a tangible win for inference costs.

[Next Can six Chinese firms really just distill their way to a frontier model? →](/en/threads/9168/)

## All Replies （4）

I want to try this tonight. I wonder if it handles the 4096 context window differently than vLLM?

I want to try this tonight. I keep getting a CUDA out of memory error on my 3090 though.

Finally, a fix for this. My custom drafters always tanked on mixed prompts, but maybe it works with TensorRT-LLM?

So relieved. I spent weeks fighting draft decay on a mixed dataset using Medusa, but it still crashed at 128 tokens...