RLVR-tuned reasoning models beat their base model at pass@1 but lose at large k: sample 128 times with a verifier and the base model solves more problems (Yue et al., 2025).
I tested the simplest fix: linear weight interpolation between the two checkpoints,
W(α) = W_base + α (W_RL − W_base)
No training, one knob, one model to serve. Pair: Qwen/Qwen2.5-7B ↔ hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo, Yue et al.'s exact protocol, six math benchmarks, pass@k up to 128 (1024 on AMC23/AIME24), math-verify grading.
Results
Which α to use
Limits: no α beats both endpoints at once, it is one 7B pair on math, and you should keep the mix in fp16 or fp32. The RLVR update is about one bf16 step of the base weights, so bf16 rounding erases most of it.
Minimal transformers version (holds both models in fp32, about 60 GB of RAM for a 7B pair; the repo script streams instead):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id, rl_id, alpha = "Qwen/Qwen2.5-7B", "hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo", 0.25
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.float32)
rl = AutoModelForCausalLM.from_pretrained(rl_id, dtype=torch.float32).state_dict()
with torch.no_grad():
for name, w in model.state_dict().items():
w.add_(alpha * (rl[name] - w)) # W_base + alpha (W_RL - W_base)
model.half().save_pretrained("mixed-a25") # fp16 or fp32, never bf16
AutoTokenizer.from_pretrained(rl_id).save_pretrained("mixed-a25")
If you try it on another base/RL pair, I would love to hear the numbers, whether it helps or not.
Paper: https://www.preprints.org/manuscript/202611.2034
Project page (animated overview, real sampled answers): Mix Your Base and RLVR Weights
Code + per-sample results: GitHub - Kim-Ai-gpu/rlvr-interp-passk: Training-free base <-> RLVR weight interpolation evaluated at large-k pass@k (Yue et al. protocol, Qwen2.5-7B <-> SimpleRL-Zoo-7B) · GitHub