# Mixing a base model with its RLVR model gets back the diversity RL takes away (training-free)

> Source: <https://discuss.huggingface.co/t/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away-training-free/192144#post_1>
> Published: 2026-11-19 11:29:57+00:00

**Recommendation:** When RLVR models underperform their base at large-k pass@k, interpolate the checkpoints at a small alpha in fp16 or fp32 and re-run pass@k up to 128 with a verifier before committing to one endpoint.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

RLVR-tuned reasoning models beat their base model at pass@1 but lose at large k: sample 128 times with a verifier and the base model solves more problems (Yue et al., 2025).

I tested the simplest fix: linear weight interpolation between the two checkpoints,

```
W(α) = W_base + α (W_RL − W_base)
```

No training, one knob, one model to serve. Pair: Qwen/Qwen2.5-7B ↔ hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo, Yue et al.'s exact protocol, six math benchmarks, pass@k up to 128 (1024 on AMC23/AIME24), math-verify grading.

Results

Which α to use

Limits: no α beats both endpoints at once, it is one 7B pair on math, and you should keep the mix in fp16 or fp32. The RLVR update is about one bf16 step of the base weights, so bf16 rounding erases most of it.

Minimal transformers version (holds both models in fp32, about 60 GB of RAM for a 7B pair; the repo script streams instead):

``` python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id, rl_id, alpha = "Qwen/Qwen2.5-7B", "hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo", 0.25
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.float32)
rl = AutoModelForCausalLM.from_pretrained(rl_id, dtype=torch.float32).state_dict()
with torch.no_grad():
    for name, w in model.state_dict().items():
        w.add_(alpha * (rl[name] - w))           # W_base + alpha (W_RL - W_base)
model.half().save_pretrained("mixed-a25")        # fp16 or fp32, never bf16
AutoTokenizer.from_pretrained(rl_id).save_pretrained("mixed-a25")
```

If you try it on another base/RL pair, I would love to hear the numbers, whether it helps or not.

Paper: [https://www.preprints.org/manuscript/202611.2034](https://www.preprints.org/manuscript/202611.2034)

Project page (animated overview, real sampled answers): [Mix Your Base and RLVR Weights](https://kim-ai-gpu.github.io/rlvr-interp-passk/)

Code + per-sample results: [GitHub - Kim-Ai-gpu/rlvr-interp-passk: Training-free base <-> RLVR weight interpolation evaluated at large-k pass@k (Yue et al. protocol, Qwen2.5-7B <-> SimpleRL-Zoo-7B) · GitHub](https://github.com/Kim-Ai-gpu/rlvr-interp-passk)
