{"slug": "r-2-po-decoupling-rollout-and-inference-policies-for-llm-reasoning", "title": "R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning", "summary": "Researchers propose R²PO, a method that decouples rollout and inference policies for LLM reasoning, achieving accuracy gains of 3.4% on MATH-500 and 1.3% on APPS. The approach attaches a lightweight Residual Rollout-Head to diversify training trajectories while preserving inference quality, outperforming baselines with reduced length bias.", "body_md": "arXiv:2601.11960v3 Announce Type: replace-cross\nAbstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R$^2$PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R$^2$PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.", "url": "https://wpnews.pro/news/r-2-po-decoupling-rollout-and-inference-policies-for-llm-reasoning", "canonical_source": "https://www.machinebrief.com/news/rdollar2dollarpo-decoupling-rollout-and-inference-policies-f-t4ka", "published_at": "2026-07-07 04:00:00+00:00", "updated_at": "2026-07-07 19:39:42.109524+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research"], "entities": ["R²PO", "MATH-500", "APPS", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/r-2-po-decoupling-rollout-and-inference-policies-for-llm-reasoning", "markdown": "https://wpnews.pro/news/r-2-po-decoupling-rollout-and-inference-policies-for-llm-reasoning.md", "text": "https://wpnews.pro/news/r-2-po-decoupling-rollout-and-inference-policies-for-llm-reasoning.txt", "jsonld": "https://wpnews.pro/news/r-2-po-decoupling-rollout-and-inference-policies-for-llm-reasoning.jsonld"}}