PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization

Researchers introduced PPO-HSC (Proximal Policy Optimization with High-order Sampling Coverage), an exploratory reinforcement learning framework that addresses mode collapse in Large Language Model fine-tuning by incentivizing discovery of low-similarity yet high-validity reasoning patterns. Evaluations on GSM8K, SVAMP, and code generation tasks show PPO-HSC enhances solution diversity and state-space coverage while maintaining or surpassing accuracy of state-of-the-art RL baselines.

arXiv:2607.16206v1 Announce Type: new Abstract: This paper introduces PPO-HSC Proximal Policy Optimization with High-order Sampling Coverage , an exploratory reinforcement learning framework designed to address the "Invisible Shackles" of mode collapse in Large Language Model LLM fine-tuning. While standard Reinforcement Learning from Verifiable Rewards RLVR effectively reinforces high-reward trajectories, it often leads models to over-optimize known solutions, sacrificing curiosity and the ability to explore broader solution manifolds. To overcome this, PPO-HSC incorporates a High-order Sampling Coverage HSC reward that incentivizes the discovery of "low-similarity yet high-validity" reasoning patterns. By maintaining a dynamic trajectory library of verified unique solutions, the framework provides a differentiable signal that rewards semantic novelty while ensuring structural rationality through a plausibility constraint. Empirical evaluations on mathematical reasoning GSM8K, SVAMP and code generation tasks demonstrate that PPO-HSC significantly enhances solution diversity and state-space coverage while maintaining or surpassing the accuracy and syntax integrity of state-of-the-art RL baselines.