{"slug": "cross-behavior-policy-optimization-unified-learning-from-offline-demonstrations", "title": "Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target", "summary": "A new arXiv paper (2610.12830v1) proposes Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and on- and off-policy rollouts under a single regression loss with a fixed-reference target, eliminating the need for separate supervised fine-tuning and reinforcement learning objectives, stored behavior-policy probabilities, importance weighting, or a learned critic. On Qwen3-1.7B across four mathematical reasoning benchmarks, CBPO-Unified improved benchmark-averaged avg@32 and pass@32 over SFT DAPO by 2.33 and 4.44 percentage points, respectively, and retaining historical rollouts in the replay ablation added 1.87 and 3.72 percentage points over the no-history configuration.", "body_md": "arXiv:2610.12830v1 Announce Type: new \nAbstract: Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose \\textbf{Cross-Behavior Policy Optimization} (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading \\texttt{avg@32} and \\texttt{pass@32} performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged \\texttt{avg@32} and \\texttt{pass@32} over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.", "url": "https://wpnews.pro/news/cross-behavior-policy-optimization-unified-learning-from-offline-demonstrations", "canonical_source": "https://www.machinebrief.com/news/cross-behavior-policy-optimization-unified-learning-from-off-5ern", "published_at": "2026-10-12 04:00:00+00:00", "updated_at": "2026-10-12 05:08:19.890529+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Cross-Behavior Policy Optimization", "CBPO", "CBPO-Unified", "Qwen3-1.7B", "SFT DAPO", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cross-behavior-policy-optimization-unified-learning-from-offline-demonstrations", "markdown": "https://wpnews.pro/news/cross-behavior-policy-optimization-unified-learning-from-offline-demonstrations.md", "text": "https://wpnews.pro/news/cross-behavior-policy-optimization-unified-learning-from-offline-demonstrations.txt", "jsonld": "https://wpnews.pro/news/cross-behavior-policy-optimization-unified-learning-from-offline-demonstrations.jsonld"}}