# DPO Fine-Tuning from First Principles in Python > Source: > Published: 2026-07-12 22:31:00+00:00 Member-only story # DPO Fine-Tuning from First Principles in Python Fine-tuning a Large Language Model (LLM) with human preferences used to require Reinforcement Learning from Human Feedback (RLHF): collect human ratings, train a reward model, then run PPO (Proximal Policy Optimization). The pipeline has four separate models, significant hyperparameter complexity, and known instability. Direct Preference Optimization (DPO) collapses all of this into a single supervised loss over preference pairs — no reward model, no RL loop. Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information. ## Why RLHF Is Hard DPO is best understood as a reaction to the problems of its predecessor. Before seeing what DPO does, it helps to understand exactly why the standard RLHF pipeline is so difficult to work with in practice. Standard RLHF: - Collect (prompt, chosen response, rejected response) triples from human raters - Train a reward model: `r_φ(x, y)` — a scalar score for response`y` to prompt`x` - Fine-tune the LLM policy `π_θ` with PPO to maximize`E[r_φ(x,y)] - β × KL[π_θ ‖ π_ref]` The KL term keeps the policy from straying too far from the reference (pre-SFT…