R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning

Researchers propose R²PO, a method that decouples rollout and inference policies for LLM reasoning, achieving accuracy gains of 3.4% on MATH-500 and 1.3% on APPS. The approach attaches a lightweight Residual Rollout-Head to diversify training trajectories while preserving inference quality, outperforming baselines with reduced length bias.

arXiv:2601.11960v3 Announce Type: replace-cross Abstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R$^2$PO Residual Rollout Policy Optimization , which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R$^2$PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.