{"slug": "mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away", "title": "Mixing a base model with its RLVR model gets back the diversity RL takes away (training-free)", "summary": "Linear weight interpolation between Qwen/Qwen2.5-7B and hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo recovers much of the pass@k diversity that RLVR training removes, according to a training-free experiment by the author of the rlvr-interp-passk project, who evaluated the mix on six math benchmarks at pass@k up to 128 (1024 on AMC23/AIME24) using math-verify grading under Yue et al.'s protocol. The author reports that no single alpha beats both endpoints at once and that the mix must be kept in fp16 or fp32, since the RLVR update is roughly one bf16 step of the base weights and bf16 rounding erases most of it. The result matters because RLVR-tuned reasoning models beat their base model at pass@1 but lose at large k, where the base model solves more problems when sampled 128 times with a verifier.", "body_md": "RLVR-tuned reasoning models beat their base model at pass@1 but lose at large k: sample 128 times with a verifier and the base model solves more problems (Yue et al., 2025).\n\nI tested the simplest fix: linear weight interpolation between the two checkpoints,\n\n```\nW(α) = W_base + α (W_RL − W_base)\n```\n\nNo training, one knob, one model to serve. Pair: Qwen/Qwen2.5-7B ↔ hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo, Yue et al.'s exact protocol, six math benchmarks, pass@k up to 128 (1024 on AMC23/AIME24), math-verify grading.\n\nResults\n\nWhich α to use\n\nLimits: no α beats both endpoints at once, it is one 7B pair on math, and you should keep the mix in fp16 or fp32. The RLVR update is about one bf16 step of the base weights, so bf16 rounding erases most of it.\n\nMinimal transformers version (holds both models in fp32, about 60 GB of RAM for a 7B pair; the repo script streams instead):\n\n``` python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nbase_id, rl_id, alpha = \"Qwen/Qwen2.5-7B\", \"hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo\", 0.25\nmodel = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.float32)\nrl = AutoModelForCausalLM.from_pretrained(rl_id, dtype=torch.float32).state_dict()\nwith torch.no_grad():\n    for name, w in model.state_dict().items():\n        w.add_(alpha * (rl[name] - w))           # W_base + alpha (W_RL - W_base)\nmodel.half().save_pretrained(\"mixed-a25\")        # fp16 or fp32, never bf16\nAutoTokenizer.from_pretrained(rl_id).save_pretrained(\"mixed-a25\")\n```\n\nIf you try it on another base/RL pair, I would love to hear the numbers, whether it helps or not.\n\nPaper: [https://www.preprints.org/manuscript/202611.2034](https://www.preprints.org/manuscript/202611.2034)\n\nProject page (animated overview, real sampled answers): [Mix Your Base and RLVR Weights](https://kim-ai-gpu.github.io/rlvr-interp-passk/)\n\nCode + per-sample results: [GitHub - Kim-Ai-gpu/rlvr-interp-passk: Training-free base <-> RLVR weight interpolation evaluated at large-k pass@k (Yue et al. protocol, Qwen2.5-7B <-> SimpleRL-Zoo-7B) · GitHub](https://github.com/Kim-Ai-gpu/rlvr-interp-passk)", "url": "https://wpnews.pro/news/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away", "canonical_source": "https://discuss.huggingface.co/t/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away-training-free/192144#post_1", "published_at": "2026-11-19 11:29:57+00:00", "updated_at": "2026-11-19 11:41:36.497563+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-safety"], "entities": ["Qwen2.5-7B", "hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo", "Yue et al.", "Hugging Face Transformers", "AMC23", "AIME24"], "also_reported_by": [], "takeaway": "Interpolating base and RLVR weights at alpha=0.25 (W_base + 0.25(W_RL − W_base)) in fp16 or fp32, never bf16, recovers large-k pass@k diversity without training, though no alpha beats both endpoints at once.", "recommendation": {"advice": "When RLVR models underperform their base at large-k pass@k, interpolate the checkpoints at a small alpha in fp16 or fp32 and re-run pass@k up to 128 with a verifier before committing to one endpoint.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away", "markdown": "https://wpnews.pro/news/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away.md", "text": "https://wpnews.pro/news/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away.txt", "jsonld": "https://wpnews.pro/news/mixing-a-base-model-with-its-rlvr-model-gets-back-the-diversity-rl-takes-away.jsonld"}}