cd /news/machine-learning/mixing-a-base-model-with-its-rlvr-mo… · home › topics › machine-learning › article
[ARTICLE · art-172493] src=discuss.huggingface.co ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Mixing a base model with its RLVR model gets back the diversity RL takes away (training-free)

Linear weight interpolation between Qwen/Qwen2.5-7B and hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo recovers much of the pass@k diversity that RLVR training removes, according to a training-free experiment by the author of the rlvr-interp-passk project, who evaluated the mix on six math benchmarks at pass@k up to 128 (1024 on AMC23/AIME24) using math-verify grading under Yue et al.'s protocol. The author reports that no single alpha beats both endpoints at once and that the mix must be kept in fp16 or fp32, since the RLVR update is roughly one bf16 step of the base weights and bf16 rounding erases most of it. The result matters because RLVR-tuned reasoning models beat their base model at pass@1 but lose at large k, where the base model solves more problems when sampled 128 times with a verifier.

recommendation

When RLVR models underperform their base at large-k pass@k, interpolate the checkpoints at a small alpha in fp16 or fp32 and re-run pass@k up to 128 with a verifier before committing to one endpoint.

more in playbook: Evals and testing for LLM apps
read1 min views1 publishedNov 19, 2026

RLVR-tuned reasoning models beat their base model at pass@1 but lose at large k: sample 128 times with a verifier and the base model solves more problems (Yue et al., 2025).

I tested the simplest fix: linear weight interpolation between the two checkpoints,

W(α) = W_base + α (W_RL − W_base)

No training, one knob, one model to serve. Pair: Qwen/Qwen2.5-7B ↔ hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo, Yue et al.'s exact protocol, six math benchmarks, pass@k up to 128 (1024 on AMC23/AIME24), math-verify grading.

Results

Which α to use

Limits: no α beats both endpoints at once, it is one 7B pair on math, and you should keep the mix in fp16 or fp32. The RLVR update is about one bf16 step of the base weights, so bf16 rounding erases most of it.

Minimal transformers version (holds both models in fp32, about 60 GB of RAM for a 7B pair; the repo script streams instead):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id, rl_id, alpha = "Qwen/Qwen2.5-7B", "hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo", 0.25
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.float32)
rl = AutoModelForCausalLM.from_pretrained(rl_id, dtype=torch.float32).state_dict()
with torch.no_grad():
    for name, w in model.state_dict().items():
        w.add_(alpha * (rl[name] - w))           # W_base + alpha (W_RL - W_base)
model.half().save_pretrained("mixed-a25")        # fp16 or fp32, never bf16
AutoTokenizer.from_pretrained(rl_id).save_pretrained("mixed-a25")

If you try it on another base/RL pair, I would love to hear the numbers, whether it helps or not.

Paper: https://www.preprints.org/manuscript/202611.2034

Project page (animated overview, real sampled answers): Mix Your Base and RLVR Weights

Code + per-sample results: GitHub - Kim-Ai-gpu/rlvr-interp-passk: Training-free base <-> RLVR weight interpolation evaluated at large-k pass@k (Yue et al. protocol, Qwen2.5-7B <-> SimpleRL-Zoo-7B) · GitHub

── more in #machine-learning 4 stories · sorted by recency
── more on @qwen2.5-7b 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/mixing-a-base-model-…] indexed:0 read:1min 2026-11-19 · —