A large language model trained on the internet is exceptionally good at one thing: predicting the next token. Feed it “The capital of Tamil Nadu is” and it completes the sentence with “Chennai.” This is pretraining, and it gives the model remarkable capabilities. It learns grammar, facts, reasoning patterns, and stylistic nuance.
But capability is not preference alignment. Consider this prompt: ”Explain quantum computing to a 10-year-old.” The model might generate:
Response A: A technically accurate explanation involving superposition, entanglement, and quantum gates. Correct, but impenetrable to a child.
Response B: A simple analogy involving spinning coins. Accessible, but it subtly misrepresents how qubits actually work.
Response C: A playful story about a cat that can be asleep and awake at the same time. Engaging, age-appropriate, and roughly correct.
All three are grammatically fluent. All three are valid next-token predictions. Yet which one is best depends entirely on what we value: accuracy, simplicity, engagement, age-appropriateness, or avoidance of misleading analogies. Pretraining alone cannot resolve this tradeoff. We need a signal that tells the model which response is preferred.
This is preference alignment. And it raises a single question:** where should that preference signal come from?**
The journey from a base model to a helpful assistant typically follows a progression: pretraining, instruction tuning, then preference optimization.
Pretraining builds general capability. Instruction tuning, usually via Supervised Fine-Tuning (SFT), teaches the model to follow prompts by training it on high-quality examples of instructions and human-written responses. SFT can absolutely improve helpfulness, harmlessness, and honesty if those properties are present in the demonstrations. But SFT learns from demonstrations: “Here is a good answer. Imitate it.” It does not explicitly learn which of several acceptable answers is preferred.
Preference optimization fills that gap. Instead of showing the model a single correct answer, we show it multiple possible answers and tell it which one is better. This requires a mechanism for collecting, encoding, and optimizing against preferences.
Before discussing the specific methods, it helps to map standard reinforcement learning terminology onto LLMs.
Policy: In reinforcement learning, a policy (denoted as π) is the agent’s strategy or rulebook. It defines the behavior of an agent by mapping perceived states of the environment to actions to be taken when in those states. The ultimate goal of training an RL algorithm is to find an optimal policy (π*) that maximizes the expected cumulative reward over time.
Reward: a numerical feedback signal sent by the environment to tell the agent how good or bad its action was.
Reward Model: A separate model trained to predict preferences. It takes a prompt-response pair and produces a scalar score. It is not the reward itself; it is a learned approximation of the reward signal.
Policy Optimization: The process of updating the LLM to generate responses that receive higher scores from the reward model.
The generic pipeline looks like this:
This framework makes both RLHF and RLAIF possible. The defining distinction is where the preference signal comes from, although the training pipeline and evaluation methodology can differ substantially.
Reinforcement Learning from Human Feedback (RLHF), also called reinforcement learning from human preferences, is the method that turned base models into conversational assistants. The idea is straightforward: humans compare model outputs and indicate which ones they prefer, and that preference data is distilled into a reward function that guides further training.
The pipeline has three stages.
Stage 1: Supervised Fine-Tuning (SFT). The base model is trained on a small, curated dataset of prompts with human-written responses. This gives the model a reasonable starting point for conversational format and style.
Stage 2: Reward Model Training. Human annotators are shown multiple responses to the same prompt and asked to rank them. These pairwise comparisons form a preference dataset, which is used to train a reward model that learns to assign scalar scores to responses such that preferred responses receive higher scores.
The standard formulation uses the Bradley-Terry model:
Here, 𝑥 is the prompt, 𝑦𝑤 is the winning response, 𝑦ₗ is the losing response, 𝑅𝜙 is the reward model, and 𝜎 is the sigmoid function. The reward model never sees a ground-truth correct answer. It only learns to mimic the statistical patterns of human judgment.
Stage 3: Reinforcement Learning (Policy Optimization). The language model generates responses, the reward model scores them, and the LLM is updated to maximize those scores. In practice, this often uses Proximal Policy Optimization (PPO). PPO constrains policy updates through its clipped objective, which discourages excessively large updates. RLHF implementations commonly add a KL divergence penalty to keep the optimized policy close to a reference model, often the SFT checkpoint. This prevents the model from drifting into gibberish that happens to trick the reward model.
Human feedback is powerful because it handles what algorithms cannot. How do you mathematically define “funny,” “helpful,” or “tactful”? You do not. You let humans rate jokes, compare explanations, and flag harmful content. RLHF excels at capturing subjective nuance, safety concerns, and culturally specific preferences.
But it is also expensive, slow, and difficult to scale. High-quality human annotation requires trained annotators, careful quality control, and resolution of disagreements between raters. Human annotators bring their own biases. And as models improve, the gap between good and great responses narrows, making human judgments noisier and more expensive to obtain.
This scalability bottleneck raises a natural question: if collecting human feedback is expensive, could another AI model provide the feedback instead?
Reinforcement Learning from AI Feedback (RLAIF) refers broadly to alignment approaches in which an AI system generates some or all of the preference or evaluation signal used during training. Instead of humans ranking responses, a separate model generates preference labels, critiques, or scores.
The pipeline is structurally similar:
The AI evaluator can operate in several modes:
It is worth distinguishing three concepts that people often conflate:
Constitutional AI, developed by Anthropic, is one of the most concrete instantiations of RLAIF. The core idea is to give the AI evaluator explicit principles for judging outputs. For example: “Does this response contain harmful content?” or “Is this response honest about uncertainty?” The target model generates a response, the evaluator critiques it against these principles, and the model is trained to produce responses that satisfy the constitution.
Constitutional AI involves a specific methodology: constitutional principles, self-critique, revision, AI-generated preference data, supervised learning, and reinforcement learning from AI feedback. RLAIF is the broader concept; Constitutional AI is one approach that uses AI-generated feedback and explicit principles.
The reward model learns to predict which response humans prefer. Given a prompt 𝑥 and two responses 𝑦𝑤 (the preferred, or “winning” response) and 𝑦ₗ (the rejected, or “losing” response), the Bradley-Terry model gives the probability that humans would choose 𝑦𝑤 over 𝑦ₗ:
If R𝜙(𝑥, 𝑦𝑤) is much larger than R𝜙(𝑥, 𝑦ₗ), the probability approaches 1. If the scores are similar, the probability approaches 0.5. Training adjusts the parameters 𝜙 so that preferred responses consistently receive higher scores. The reward model is trained to maximize the probability assigned to the human-preferred response. The loss function is:
When the preferred response receives a lower reward than the rejected response, the loss increases and the model updates. This connects human preference to a concrete mathematical objective.
The language model policy π𝜃 is optimized to generate responses that maximize the reward model’s score, while a KL divergence penalty keeps it from drifting too far from the reference policy (typically the SFT model):
The first term encourages high-reward responses. The second term penalizes large deviations from the reference, preventing the model from collapsing into outputs that merely exploit the reward model.
DPO bypasses the explicit reward model and RL loop, optimizing the policy directly from preference data:
DPO encourages the policy to assign relatively higher probability to preferred responses than to rejected ones, relative to a reference model. This gives a direct path from preferences to policy updates without the intermediate reward model training stage.
If you run an experiment comparing AI evaluator judgments to human judgments, simple statistics quantify the agreement. The raw agreement rate is: A more rigorous metric that accounts for chance agreement is Cohen’s kappa:
where pₒ is the observed agreement and pₑ is the agreement expected by chance. For example, an AI evaluator that agrees with humans on 82% of comparisons with a Cohen’s κ of 0.61 indicates substantial agreement beyond random chance.
At first glance, the tradeoff seems simple: humans provide nuanced but expensive feedback; AI provides scalable but potentially shallow feedback. But the reality is more subtle.
The table is useful, but it risks oversimplification. The crucial insight is that the quality of the feedback matters more than whether the feedback is technically human or AI-generated. A well-designed AI evaluator with carefully crafted rubrics can outperform tired, inconsistent human annotators. Conversely, a poorly designed AI evaluator can systematically reinforce its own blind spots.
And here is the deeper problem: *an evaluator can produce 10 million labels very cheaply while systematically making the same mistake 10 million times. *Scale of supervision does not equal quality of supervision.
With RLHF, the chain of accountability is clear:
With RLAIF, it becomes recursive:
This raises a deeper set of concerns that go beyond simple cost comparisons.
This is fundamentally a problem of proxy objectives. The human goal is to produce helpful, accurate, safe responses. The proxy is the reward model score. The model optimizes the proxy. When a proxy measure becomes the target, optimizing the proxy can cause behavior that diverges from the original objective. This is Goodhart’s Law, and it is almost the philosophical foundation of reward-model-based alignment.
There is also the problem of evaluator-target similarity. If the evaluator and target model share similar training data and architecture, they may share correlated weaknesses. A model evaluating itself, or a sibling model evaluating a near-twin, may fail to catch errors that a human would immediately notice. The blind spots reinforce each other.
And then there is reward model overoptimization. As policy optimization becomes stronger, the model can become increasingly good at maximizing reward-model scores without producing proportionally better outputs. At some point, the reward score keeps climbing while actual human quality plateaus or even declines.
Rather than declaring a winner, it is more honest to think in terms of fit.
Human feedback has an advantage when:
AI feedback has an advantage when:
Neither is sufficient when evaluation requires knowledge that neither the annotator nor the evaluator reliably possesses. In those cases, you may need expert evaluation or external verification.
In practice, the choice is rarely binary.
A compelling practical direction is a hybrid system in which humans and AI work together.
Humans focus on what they do best: difficult judgment calls, edge cases, safety-critical decisions, and evaluating whether the AI evaluator itself is trustworthy. AI handles the high-volume, routine work: generating synthetic preferences, performing pairwise comparisons at scale, and producing structured critiques.
This division of labor preserves human oversight where it matters most while leveraging AI scalability where it is genuinely useful. But even hybrid systems need independent validation. You should never evaluate a model solely with the same evaluator that trained it. Otherwise you risk circular reasoning: the judge trains the model, the judge evaluates the model, and the judge concludes the model improved.
Consider again: ”Explain quantum computing to a 10-year-old.”
Response A is technically accurate but uses terms like “superposition” without explanation. Response B uses a simple spinning-coin analogy. Accessible, but incorrect in a subtle way. Response C uses a playful cat story. Engaging and roughly correct.
A human evaluator might produce ratings like this:
The alignment problem is not simply identifying a correct answer. It is deciding how these competing objectives should be weighted. A reward model learns from these preferences. Over many examples, the policy learns to generate explanations that balance simplicity and accuracy, or whatever tradeoff the preference data encodes.
But if the evaluator is flawed, the policy learns something else. If the evaluator prefers confident phrasing, the model becomes confidently wrong. If the evaluator prefers longer responses, the model becomes verbose. The quality of the final model depends entirely on the quality of the feedback signal.
RLHF is often used as shorthand for preference-based alignment, but not every preference-learning method requires PPO and an explicit reward model. Direct Preference Optimization (DPO) optimizes the policy directly from preference data without training a separate reward model or running a full RL loop. This distinction matters when comparing modern alignment pipelines. RLHF and RLAIF describe where the preference signal comes from; DPO describes one way to optimize from that signal.
The evolution of LLM alignment traces a clear arc: from human-written examples, to human preferences, to AI-generated feedback, to AI-assisted evaluation, to more scalable supervision. Each step addresses the scalability bottleneck of the previous one. But each step also introduces a new form of indirection. When humans supervise directly, their values are present in the training signal. When AI supervises, human values are present only to the extent that the AI evaluator correctly encodes them.
This raises the central question: if increasingly capable AI systems can supervise other AI systems, how do we ensure that the supervision remains grounded in human values?
RLAIF does not eliminate the alignment problem. It moves part of the problem upstream, to the design of the evaluator. The evaluator becomes the new frontier. If we get the evaluator wrong, everything downstream is wrong, no matter how sophisticated the RL algorithm.
If you are building one of these systems, you do not need to write everything from scratch. Here is what the tooling looks like in practice. A few notes from experience. TRL covers most of the training stages well enough that you can get a full RLHF loop running without writing your own PPO implementation. For preference data collection, Argilla works well if you want an open source annotation server, but many teams just build a simple Gradio interface because the ranking task is simple enough. For inference, vLLM is hard to beat if you are generating millions of responses for an AI evaluator. And if you are running RLAIF, your evaluator is usually just another LLM API call or a locally hosted model, so the real work is in designing the prompt and rubric, not in the tool itself.
-
RLHF uses human preferences to align models. It is powerful for capturing nuance but expensive to scale.
-
RLAIF uses AI-generated feedback to make alignment more scalable. It trades annotation cost for evaluator design cost.
-
RLAIF does not solve the alignment problem. It relocates it. The evaluator becomes the critical bottleneck.
-
Both approaches have failure modes: human bias and inconsistency on one side, evaluator bias and reward hacking on the other.
-
The bottleneck in RLAIF is not generating feedback. It is knowing whether the feedback is worth trusting.
The future of AI alignment may not be about choosing who teaches AI, humans or machines, but about designing systems where machines can scale human judgment without quietly replacing it.
If you enjoyed reading this, I would like to hear your feedback. Please clap or comment and follow me. Thanks!
-
Christiano, P. F. et al. (2017). Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741.
-
Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
-
Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155.
-
Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
-
Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.
RLHF vs RLAIF: Who Should Teach an AI What “Good” Looks Like? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.