{"slug": "predictable-grpo-a-closed-form-model-of-training-dynamics", "title": "Predictable GRPO: A Closed-Form Model of Training Dynamics", "summary": "Researchers developed a closed-form model of Group Relative Policy Optimization (GRPO) training dynamics, predicting reward trajectories and stability thresholds from first principles. The model subsumes empirical saturation laws, yields group-size invariance and oscillatory transitions, and diagnoses failure modes like reward hacking. Across models and benchmarks, the closed-form fits achieved R² ≥ 0.91.", "body_md": "arXiv:2606.30789v1 Announce Type: new\nAbstract: Group Relative Policy Optimization (GRPO) has become a standard tool for improving the reasoning ability of large language models, yet its training dynamics are still described empirically: reward trajectories are fit with low-parameter functional forms whose constants carry no mechanistic meaning, and hyperparameter choices remain a matter of trial and error. We develop a first-principles reduced-order model of these dynamics. The reduction has three consequences. First, it subsumes the empirical single-exponential saturation law as its overdamped limit, recasting the fitted plateau, timescale, and size exponent as the fixed point, inverse stiffness, and curvature-scaling exponent of the underlying potential, and adding, through the retained inertial term, the slow-start phase the single exponential cannot represent. Second, it yields predictions tied to independently measurable quantities rather than fitted ones: group-size invariance of the deterministic trajectory with a $1/G$ stationary fluctuation, a sharp stability threshold in the refresh interval, and an overdamped-to-oscillatory transition. Third, it furnishes diagnostics that separate failure modes a reward curve alone conflates -- reward hacking, advantage degeneracy, policy concentration, and dynamical instability. Across three models and two group sizes, the closed-form trajectory fits training reward to $R^2 \\geq 0.91$ and the predicted group-size invariance holds on both the reward curve and out-of-distribution transfer to eight math benchmarks. The stability and oscillatory predictions are exercised in a controlled exact-reduction setting where the mean-field assumption holds exactly: a softmax-bandit reduction reproduces the predicted overdamped-to-oscillatory transition and locates the refresh-interval stability threshold at the independently measured stiffness, with a deep-network demonstration left to future work.", "url": "https://wpnews.pro/news/predictable-grpo-a-closed-form-model-of-training-dynamics", "canonical_source": "https://arxiv.org/abs/2606.30789", "published_at": "2026-07-01 04:00:00+00:00", "updated_at": "2026-07-01 04:30:32.388675+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "ai-research"], "entities": ["GRPO", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/predictable-grpo-a-closed-form-model-of-training-dynamics", "markdown": "https://wpnews.pro/news/predictable-grpo-a-closed-form-model-of-training-dynamics.md", "text": "https://wpnews.pro/news/predictable-grpo-a-closed-form-model-of-training-dynamics.txt", "jsonld": "https://wpnews.pro/news/predictable-grpo-a-closed-form-model-of-training-dynamics.jsonld"}}