SimHack: Predicting the onset of detectable reward hacking before RL training

A new arXiv paper (2610.28707v1) introduces SimHack, a simulator that predicts when detectable reward hacking will become dominant in an LLM coding RL run before training begins, learning the dynamics from a handful of smaller-scale reference runs. Across unseen target recipes, SimHack predicts the median reward hacking onset step to a relative error of 11.3% and achieves an across-recipe rank correlation of 0.85, generalizing across model scale, curriculum, number of rollouts and learning rate. As an application, SimHack screened candidate RL curricula and found curricula over the same scenarios that train 96.6% longer without hacking.

arXiv:2610.28707v1 Announce Type: new Abstract: When training LLMs to code with reinforcement learning, reward hacking is often detectable but difficult to prevent, since a environment hardening is increasingly difficult as training environments grow more complex; and b training-time interventions incentivize models to obfuscate hacking behavior to bypass detection. We instead study whether we can predict when detectable reward hacking becomes dominant before executing an RL run, enabling us to select lower-risk recipes in advance. We find that the initial hacking rate alone does not determine onset, which varies substantially across training recipes and repeated runs. We thus formulate this problem as predicting the distribution of the reward hacking onset step under a target training recipe. We introduce SimHack, a simulator that learns reward hacking training dynamics from a handful of smaller-scale reference RL runs and generates counterfactual simulation runs of unseen target recipes without costly retraining. Across unseen target recipes, SimHack predicts the median onset to a relative error of 11.3% and achieves across-recipe rank correlation of 0.85, while generalizing to differences in model scale, curriculum, number of rollouts, and learning rate between reference and target recipes. As an application, we show that SimHack can guide training decisions by screening candidate RL curricula, discovering curricula over the same set of scenarios that train for 96.6% longer without hacking.