Your AI Knows How to Answer. But Who Teaches It What a Good Answer Is?

Rijul, a developer building the AI code review tool LiveReview, published an explainer comparing two methods for aligning language models with human preferences: Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF). The piece walks through how DPO learns directly from chosen-versus-rejected answer pairs, while RLHF trains a separate reward model on human comparisons and then uses reinforcement learning to update the language model. It notes that both approaches are only as good as the preference data behind them, warning that noisy, inconsistent, or biased human judgments can propagate into the model.

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us https://github.com/HexmosTech/LiveReview/ to help devs discover the project, give it a try, and share your feedback to help improve the product. You use ChatGPT, Gemini, and all these AI bots. Whatever you ask, they usually give you a good answer, often something we like to hear. Ever wondered how they are tuned to our tastes? We humans have a hand in that too. We teach these models to produce responses that people prefer. There are different ways to teach a language model to produce responses that people prefer. Two important approaches are DPO and RLHF . Both use human preferences to guide a model, but they do it differently. Let's understand how each one works. DPO stands for Direct Preference Optimization . It is a way to teach a language model which kinds of answers people prefer. When a model learns to generate text, you usually want it to produce responses that are helpful, follow instructions, and avoid undesirable behavior. But describing exactly what makes an answer "good" can be difficult. Instead, we can ask humans to compare answers. For example, given the same question, the model might produce two responses: Answer A: Clear, helpful, and directly answers the question. Answer B: Long and confusing, and doesn't really answer the question. A human can simply say: A is better than B. DPO uses many of these preferences to adjust the model so that it becomes more likely to produce responses similar to the preferred answers. You provide training examples containing: For example: Prompt: How do I reset my password? Chosen: Go to Settings → Security → Reset Password and follow the instructions. Rejected: Passwords can be changed in many different ways depending on the situation. The model learns from many such comparisons. A simple way to think about it is: "When you see situations like this, produce something more like A and less like B." After seeing enough comparisons, the model learns patterns in the preferences. The model learns from the preference data it receives. If the human judgments are noisy, inconsistent, or biased, those problems can make their way into the model. For example, if humans consistently prefer overly long answers, the model may learn that longer answers are better even when they are not. So DPO is only as good as the preference data used to train it. Now let's look at another approach. RLHF stands for Reinforcement Learning from Human Feedback . Like DPO, RLHF uses human preferences to teach a model which responses people prefer. But the training process is different. A classic RLHF pipeline has three major stages. First, you start with a pretrained language model. It already knows how to generate text, but it may not reliably follow instructions. So it is usually fine-tuned on examples of good conversations and instructions. This gives you a model that can follow instructions reasonably well. Now humans compare different answers from the model. Question: Explain photosynthesis. Answer A: Photosynthesis is the process plants use to convert light energy into chemical energy. Answer B: Plants use sunlight, water, and carbon dioxide in a biological process. Humans indicate which answer they prefer. A separate model, called a reward model , is then trained on many of these preferences. Its job is to predict how much a human would prefer a particular response. Instead of simply saying: A is better than B the reward model can assign scores to responses. Answer A → 0.82 Answer B → 0.54 The reward model has learned to act as a rough approximation of the human preferences represented in its training data. Now the language model generates new responses. The reward model scores those responses. The language model is then updated using reinforcement learning to produce responses that receive higher rewards. You can think of the process like this: Language model ↓ Generates an answer ↓ Reward model ↓ Gives a score ↓ Reinforcement learning ↓ Updates the language model There is also a constraint that keeps the updated model from moving too far away from its original behavior. In classic RLHF, this is commonly implemented using a KL-divergence penalty. Without such constraints, the model could find strange ways to increase its reward without actually producing better responses. Imagine a student learning to write essays. First, a group of teachers evaluates a collection of essays and decides which ones are better. Then, a grading assistant is trained to imitate those teachers' judgments. Now the student can write a new essay, receive a score from the grading assistant, and use that feedback to improve. The teachers don't need to grade every essay themselves. The grading assistant provides feedback repeatedly while the student practices. That is roughly the idea behind the reward-model stage of RLHF. The biggest difference is what happens after humans provide their preferences. With RLHF , those preferences are first used to train a separate reward model. The language model then uses reinforcement learning to optimize against that reward model. With DPO , there is no separate reward-model-and-RL loop in the standard DPO procedure. The preference pairs are used directly to optimize the language model. You can think of the difference like this: RLHF Human preferences ↓ Reward model ↓ Reinforcement learning ↓ Language model DPO Human preferences ↓ Direct preference optimization ↓ Language model This makes DPO's training pipeline simpler than the classic RLHF setup. RLHF gives you an explicit reward model and an RL optimization process, but that also makes the training pipeline more complicated. There are more moving parts, and reinforcement learning introduces its own training challenges. DPO removes the separate reward-model training and RL optimization loop, making the preference-training process simpler. But DPO still depends heavily on the quality of the preference data. In both approaches, there is a fundamental limitation: The model can only learn the preferences that are represented in the feedback it receives. If the human preferences are inconsistent, biased, or poorly defined, the resulting model can inherit those problems. DPO and RLHF are two different ways of using human preferences to shape the behavior of a language model. RLHF uses a reward model and reinforcement learning to turn human preferences into a training signal. DPO uses preference comparisons directly to optimize the model. The important difference is not that one uses human preferences and the other doesn't. Both do. The difference is how those preferences are turned into updates to the language model. Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview , a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius : how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. | The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | |---|---|---| Here's the goal: Click below to try LiveReview with your codebase: