{"slug": "theoretical-limits-of-language-model-alignment", "title": "Theoretical Limits of Language Model Alignment", "summary": "A November 2026 paper by Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald and Federico Danieli derives a closed-form expression for the maximum achievable expected reward gain under a fixed KL-divergence budget in language model alignment, governed by a Jeffreys divergence term rather than the KL term used in prior analyses. The authors reformulate the expression as a covariance under the base model, yielding an estimator that predicts achievable alignment gains from base model samples alone, and prove that reward ensembling mitigates reward hacking while showing the ideal-to-proxy gap grows with reward error and shrinks as the KL penalty factor decreases. Empirically, on safety and summarization tasks, best-of-n alignment closely approaches the theoretical KL–reward Pareto frontier while PPO and GRPO remain substantially suboptimal, leading the authors to conclude that algorithmic improvements are needed to achieve optimal alignment without high inference costs.", "body_md": "[content type paper](https://machinelearning.apple.com/research/)published November 2026\n\nTheoretical Limits of Language Model Alignment\n\nAuthorsLucas Monteiro Paes*, Natalie Mackraz*, Barry-John Theobald, Federico Danieli\n\nLanguage model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of- alignment, which selects the highest-reward output among  independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum *achievable* expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the  used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (*reward hacking*) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL–reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of- closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.\n\nSALSA: Soup-Based Alignment Learning for Stronger Adaptation in RLHF\n\nDecember 5, 2024[research area Speech and Natural Language Processing](https://machinelearning.apple.com/research/?domain=Speech%20and%20Natural%20Language%20Processing)[Workshop at NeurIPS](https://machinelearning.apple.com/research/?event=NeurIPS%20Workshop)\n\nThis paper was accepted at the Fine-Tuning in Modern Machine Learning (FITML) Workshop at NeurIPS 2024.\n\nIn Large Language Model (LLM) development, Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning models with human values and preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence between the current policy and a frozen initial policy as a reference, which is added as a penalty in policy…\n\nOn the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization\n\nOctober 9, 2024[research area Methods and Algorithms](https://machinelearning.apple.com/research/?domain=Methods%20and%20Algorithms), [research area Speech and Natural Language Processing](https://machinelearning.apple.com/research/?domain=Speech%20and%20Natural%20Language%20Processing)[conference EMNLP](https://machinelearning.apple.com/research/?event=EMNLP)\n\nReinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1) training an explicit reward model as in RLHF, and 2) using an implicit reward learned from preference data through methods such as Direct Preference Optimization (DPO). Prior work has shown…", "url": "https://wpnews.pro/news/theoretical-limits-of-language-model-alignment", "canonical_source": "https://machinelearning.apple.com/research/theoretical-limits-alignment", "published_at": "2026-11-12 00:00:00+00:00", "updated_at": "2026-11-12 17:50:53.884360+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "machine-learning", "ai-safety", "artificial-intelligence"], "entities": ["Lucas Monteiro Paes", "Natalie Mackraz", "Barry-John Theobald", "Federico Danieli", "PPO", "GRPO"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/theoretical-limits-of-language-model-alignment", "markdown": "https://wpnews.pro/news/theoretical-limits-of-language-model-alignment.md", "text": "https://wpnews.pro/news/theoretical-limits-of-language-model-alignment.txt", "jsonld": "https://wpnews.pro/news/theoretical-limits-of-language-model-alignment.jsonld"}}