Theoretical Limits of Language Model Alignment

A November 2026 paper by Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald and Federico Danieli derives a closed-form expression for the maximum achievable expected reward gain under a fixed KL-divergence budget in language model alignment, governed by a Jeffreys divergence term rather than the KL term used in prior analyses. The authors reformulate the expression as a covariance under the base model, yielding an estimator that predicts achievable alignment gains from base model samples alone, and prove that reward ensembling mitigates reward hacking while showing the ideal-to-proxy gap grows with reward error and shrinks as the KL penalty factor decreases. Empirically, on safety and summarization tasks, best-of-n alignment closely approaches the theoretical KL–reward Pareto frontier while PPO and GRPO remain substantially suboptimal, leading the authors to conclude that algorithmic improvements are needed to achieve optimal alignment without high inference costs.

content type paper https://machinelearning.apple.com/research/ published November 2026 Theoretical Limits of Language Model Alignment AuthorsLucas Monteiro Paes , Natalie Mackraz , Barry-John Theobald, Federico Danieli Language model LM alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are i reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and ii best-of- alignment, which selects the highest-reward output among independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment reward hacking grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL–reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of- closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs. SALSA: Soup-Based Alignment Learning for Stronger Adaptation in RLHF December 5, 2024 research area Speech and Natural Language Processing https://machinelearning.apple.com/research/?domain=Speech%20and%20Natural%20Language%20Processing Workshop at NeurIPS https://machinelearning.apple.com/research/?event=NeurIPS%20Workshop This paper was accepted at the Fine-Tuning in Modern Machine Learning FITML Workshop at NeurIPS 2024. In Large Language Model LLM development, Reinforcement Learning from Human Feedback RLHF is crucial for aligning models with human values and preferences. RLHF traditionally relies on the Kullback-Leibler KL divergence between the current policy and a frozen initial policy as a reference, which is added as a penalty in policy… On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization October 9, 2024 research area Methods and Algorithms https://machinelearning.apple.com/research/?domain=Methods%20and%20Algorithms , research area Speech and Natural Language Processing https://machinelearning.apple.com/research/?domain=Speech%20and%20Natural%20Language%20Processing conference EMNLP https://machinelearning.apple.com/research/?event=EMNLP Reinforcement Learning from Human Feedback RLHF is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1 training an explicit reward model as in RLHF, and 2 using an implicit reward learned from preference data through methods such as Direct Preference Optimization DPO . Prior work has shown…