{"slug": "your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is", "title": "Your AI Knows How to Answer. But Who Teaches It What a Good Answer Is?", "summary": "Rijul, a developer building the AI code review tool LiveReview, published an explainer comparing two methods for aligning language models with human preferences: Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF). The piece walks through how DPO learns directly from chosen-versus-rejected answer pairs, while RLHF trains a separate reward model on human comparisons and then uses reinforcement learning to update the language model. It notes that both approaches are only as good as the preference data behind them, warning that noisy, inconsistent, or biased human judgments can propagate into the model.", "body_md": "*Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nYou use ChatGPT, Gemini, and all these AI bots.\n\nWhatever you ask, they usually give you a good answer, often something we like to hear.\n\nEver wondered how they are tuned to our tastes?\n\nWe humans have a hand in that too. We teach these models to produce responses that people prefer.\n\nThere are different ways to teach a language model to produce responses that people prefer.\n\nTwo important approaches are **DPO** and **RLHF**.\n\nBoth use human preferences to guide a model, but they do it differently.\n\nLet's understand how each one works.\n\nDPO stands for **Direct Preference Optimization**.\n\nIt is a way to teach a language model which kinds of answers people prefer.\n\nWhen a model learns to generate text, you usually want it to produce responses that are helpful, follow instructions, and avoid undesirable behavior.\n\nBut describing exactly what makes an answer \"good\" can be difficult.\n\nInstead, we can ask humans to compare answers.\n\nFor example, given the same question, the model might produce two responses:\n\n**Answer A:** Clear, helpful, and directly answers the question.\n\n**Answer B:** Long and confusing, and doesn't really answer the question.\n\nA human can simply say:\n\nA is better than B.\n\nDPO uses many of these preferences to adjust the model so that it becomes more likely to produce responses similar to the preferred answers.\n\nYou provide training examples containing:\n\nFor example:\n\n```\nPrompt:\nHow do I reset my password?\n\nChosen:\nGo to Settings → Security → Reset Password and follow the instructions.\n\nRejected:\nPasswords can be changed in many different ways depending on the situation.\n```\n\nThe model learns from many such comparisons.\n\nA simple way to think about it is:\n\n**\"When you see situations like this, produce something more like A and less like B.\"**\n\nAfter seeing enough comparisons, the model learns patterns in the preferences.\n\nThe model learns from the preference data it receives.\n\nIf the human judgments are noisy, inconsistent, or biased, those problems can make their way into the model.\n\nFor example, if humans consistently prefer overly long answers, the model may learn that longer answers are better even when they are not.\n\nSo DPO is only as good as the preference data used to train it.\n\nNow let's look at another approach.\n\nRLHF stands for **Reinforcement Learning from Human Feedback**.\n\nLike DPO, RLHF uses human preferences to teach a model which responses people prefer.\n\nBut the training process is different.\n\nA classic RLHF pipeline has three major stages.\n\nFirst, you start with a pretrained language model.\n\nIt already knows how to generate text, but it may not reliably follow instructions.\n\nSo it is usually fine-tuned on examples of good conversations and instructions.\n\nThis gives you a model that can follow instructions reasonably well.\n\nNow humans compare different answers from the model.\n\n```\nQuestion:\nExplain photosynthesis.\n\nAnswer A:\nPhotosynthesis is the process plants use to convert light energy into chemical energy.\n\nAnswer B:\nPlants use sunlight, water, and carbon dioxide in a biological process.\n```\n\nHumans indicate which answer they prefer.\n\nA separate model, called a **reward model**, is then trained on many of these preferences.\n\nIts job is to predict how much a human would prefer a particular response.\n\nInstead of simply saying:\n\nA is better than B\n\nthe reward model can assign scores to responses.\n\n```\nAnswer A → 0.82\nAnswer B → 0.54\n```\n\nThe reward model has learned to act as a rough approximation of the human preferences represented in its training data.\n\nNow the language model generates new responses.\n\nThe reward model scores those responses.\n\nThe language model is then updated using reinforcement learning to produce responses that receive higher rewards.\n\nYou can think of the process like this:\n\n```\nLanguage model\n      ↓\nGenerates an answer\n      ↓\nReward model\n      ↓\nGives a score\n      ↓\nReinforcement learning\n      ↓\nUpdates the language model\n```\n\nThere is also a constraint that keeps the updated model from moving too far away from its original behavior. In classic RLHF, this is commonly implemented using a KL-divergence penalty.\n\nWithout such constraints, the model could find strange ways to increase its reward without actually producing better responses.\n\nImagine a student learning to write essays.\n\nFirst, a group of teachers evaluates a collection of essays and decides which ones are better.\n\nThen, a grading assistant is trained to imitate those teachers' judgments.\n\nNow the student can write a new essay, receive a score from the grading assistant, and use that feedback to improve.\n\nThe teachers don't need to grade every essay themselves.\n\nThe grading assistant provides feedback repeatedly while the student practices.\n\nThat is roughly the idea behind the reward-model stage of RLHF.\n\nThe biggest difference is what happens after humans provide their preferences.\n\nWith **RLHF**, those preferences are first used to train a separate reward model. The language model then uses reinforcement learning to optimize against that reward model.\n\nWith **DPO**, there is no separate reward-model-and-RL loop in the standard DPO procedure. The preference pairs are used directly to optimize the language model.\n\nYou can think of the difference like this:\n\n```\nRLHF\n\nHuman preferences\n       ↓\nReward model\n       ↓\nReinforcement learning\n       ↓\nLanguage model\nDPO\n\nHuman preferences\n       ↓\nDirect preference optimization\n       ↓\nLanguage model\n```\n\nThis makes DPO's training pipeline simpler than the classic RLHF setup.\n\nRLHF gives you an explicit reward model and an RL optimization process, but that also makes the training pipeline more complicated.\n\nThere are more moving parts, and reinforcement learning introduces its own training challenges.\n\nDPO removes the separate reward-model training and RL optimization loop, making the preference-training process simpler.\n\nBut DPO still depends heavily on the quality of the preference data.\n\nIn both approaches, there is a fundamental limitation:\n\n**The model can only learn the preferences that are represented in the feedback it receives.**\n\nIf the human preferences are inconsistent, biased, or poorly defined, the resulting model can inherit those problems.\n\nDPO and RLHF are two different ways of using human preferences to shape the behavior of a language model.\n\nRLHF uses a **reward model and reinforcement learning** to turn human preferences into a training signal.\n\nDPO uses **preference comparisons directly** to optimize the model.\n\nThe important difference is not that one uses human preferences and the other doesn't.\n\n**Both do.**\n\nThe difference is **how those preferences are turned into updates to the language model.**\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub: \n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is", "canonical_source": "https://dev.to/rijultp/your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is-1fc7", "published_at": "2026-09-20 18:53:16+00:00", "updated_at": "2026-09-20 18:54:22.643623+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence", "ai-safety"], "entities": ["Rijul", "LiveReview", "HexmosTech", "ChatGPT", "Gemini", "DPO", "RLHF"], "alternates": {"html": "https://wpnews.pro/news/your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is", "markdown": "https://wpnews.pro/news/your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is.md", "text": "https://wpnews.pro/news/your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is.txt", "jsonld": "https://wpnews.pro/news/your-ai-knows-how-to-answer-but-who-teaches-it-what-a-good-answer-is.jsonld"}}