Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

A new arXiv paper (2610.12830v1) proposes Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and on- and off-policy rollouts under a single regression loss with a fixed-reference target, eliminating the need for separate supervised fine-tuning and reinforcement learning objectives, stored behavior-policy probabilities, importance weighting, or a learned critic. On Qwen3-1.7B across four mathematical reasoning benchmarks, CBPO-Unified improved benchmark-averaged avg@32 and pass@32 over SFT DAPO by 2.33 and 4.44 percentage points, respectively, and retaining historical rollouts in the replay ablation added 1.87 and 3.72 percentage points over the no-history configuration.

arXiv:2610.12830v1 Announce Type: new Abstract: Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning SFT followed by reinforcement learning RL separates their objectives and limits continued reuse of collected responses. We propose \textbf{Cross-Behavior Policy Optimization} CBPO , which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading \texttt{avg@32} and \texttt{pass@32} performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged \texttt{avg@32} and \texttt{pass@32} over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.