Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning

A new game-theoretic framework for reinforcement learning fine-tuning, proposed in arXiv:2607.26358, treats the trade-off between reward and policy drift as a sequential game where a monitor tests for deviations from a reference policy, yielding an optimal KL-regularization coefficient via concave-convex fractional programming. Experiments with Qwen3-8B and Llama-3.2-1B show competitive reward-retention trade-offs in continual learning and enable auditing of API providers serving open-source models.

arXiv:2607.26358v1 Announce Type: new Abstract: Reinforcement learning RL fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient. In practice, the coefficient is typically chosen heuristically or via hyperparameter search, which can lead to unnecessary overhead in training cost or undesirable reward-retention trade-offs. We instead propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation. Specifically, we study a sequential game in which an agent chooses a policy to maximize cumulative reward while a monitor observes policy outputs over time and tests for deviations from the reference policy. Although not originating from the same perspective, we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL-regularized RL problem for an optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability. Drawing on classical results from concave-convex fractional programming, we provide a principled method for learning this equilibrium coefficient via reduction to the KL-regularized RL objective, thus allowing for flexible integration into standard fine-tuning pipelines. In experiments with Qwen3-8B and Llama-3.2-1B, we demonstrate that our methods result in competitive reward-retention trade-offs in a continual learning setting, and illustrate how our framework may be used to audit API providers serving open-source models.