{"slug": "bv-blend-uncertainty-weighted-historical-baselines-for-stable-critic-free-rl", "title": "BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards", "summary": "Researchers introduced BV-Blend, a critic-free reinforcement learning framework that stabilizes advantage estimation for aligning large language models by blending prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. The method addresses instability in GRPO-style approaches when all rollouts in a prompt group receive identical rewards, improving training stability and performance on verifiable reasoning benchmarks.", "body_md": "arXiv:2606.28707v1 Announce Type: new\nAbstract: Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models. However, GRPO-style advantage estimation depends on prompt-local (within-prompt-group) reward statistics and can be unstable. In particular, when all rollouts in a prompt group receive identical rewards, the within-group reward variance becomes zero, and group normalization yields zero advantages for that group, impeding learning in cold-start regimes with binary verifiers. We introduce BV-Blend, a critic-free framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. BV-Blend maintains EMA-tracked reward moments for each cluster, derives a confidence weight from a standard error of the mean (SEM) proxy, and uses this weight to blend historical and prompt-local baseline and variance statistics into a standardized advantage for PPO-style clipped updates. Experiments on verifiable reasoning benchmarks show that BV-Blend improves training stability and performance, and remains robust in regimes where group-normalized methods may stall.", "url": "https://wpnews.pro/news/bv-blend-uncertainty-weighted-historical-baselines-for-stable-critic-free-rl", "canonical_source": "https://arxiv.org/abs/2606.28707", "published_at": "2026-06-30 04:00:00+00:00", "updated_at": "2026-06-30 04:31:20.579227+00:00", "lang": "en", "topics": ["large-language-models", "ai-research"], "entities": ["BV-Blend", "Group Relative Policy Optimization", "GRPO", "PPO"], "alternates": {"html": "https://wpnews.pro/news/bv-blend-uncertainty-weighted-historical-baselines-for-stable-critic-free-rl", "markdown": "https://wpnews.pro/news/bv-blend-uncertainty-weighted-historical-baselines-for-stable-critic-free-rl.md", "text": "https://wpnews.pro/news/bv-blend-uncertainty-weighted-historical-baselines-for-stable-critic-free-rl.txt", "jsonld": "https://wpnews.pro/news/bv-blend-uncertainty-weighted-historical-baselines-for-stable-critic-free-rl.jsonld"}}