Member-only story
Fine-tuning a Large Language Model (LLM) with human preferences used to require Reinforcement Learning from Human Feedback (RLHF): collect human ratings, train a reward model, then run PPO (Proximal Policy Optimization). The pipeline has four separate models, significant hyperparameter complexity, and known instability. Direct Preference Optimization (DPO) collapses all of this into a single supervised loss over preference pairs — no reward model, no RL loop.
Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information.
Why RLHF Is Hard #
DPO is best understood as a reaction to the problems of its predecessor. Before seeing what DPO does, it helps to understand exactly why the standard RLHF pipeline is so difficult to work with in practice.
Standard RLHF:
- Collect (prompt, chosen response, rejected response) triples from human raters
- Train a reward model:
`r_φ(x, y)`
— a scalar score for responsey
to promptx
- Fine-tune the LLM policy
π_θ
with PPO to maximizeE[r_φ(x,y)] - β × KL[π_θ ‖ π_ref]
The KL term keeps the policy from straying too far from the reference (pre-SFT…