# SFT, RL and DPO: The Other Stack > Source: > Published: 2026-08-29 19:01:01+00:00 *A bonus chapter, not part eight. Everything below the arrow is the rest of this series.* This series has been about serving a model. What it costs, where the time goes, which settings matter. All of it assumed the model already existed. Somebody handed you a set of weights and your job started there. Post-training is where those weights got their behaviour. SFT, DPO, PPO, GRPO and RLVR are the methods, and it is a different job from serving, done with different tools and usually by different people. Most of the time post-training and serving never touch, which is fine. Reinforcement learning is the exception. It makes its own training data by running the model. That means a training run is full of inference, and the speed of your serving stack decides how long the training takes. **Pre-training** is the enormous, expensive part: read most of the internet, learn to predict the next token. What comes out completes text and is close to useless as an assistant. **Post-training** turns that into something you would ship. It is comparatively cheap, and almost all the behaviour you care about comes from here. When people talk about fine-tuning their own model, this is usually what they mean. Then serving, which is the other seven parts. *Pre-training stops at the middle box. Post-training picks it up from there.* **Supervised fine-tuning** is the plain one. Collect examples of the behaviour you want, a prompt and the response you wish the model had given, and train on them directly. Same objective as pre-training, but on curated data. It teaches format, instruction-following, tone and domain register. It is cheap, well understood, and **still where most of the practical value is**. LoRA (low-rank adaptation) is how most people afford it. It freezes the base weights and trains a small pair of matrices alongside them, so what you update is a fraction of the model. The saving is not really the parameter count. A frozen weight carries no gradient and no optimizer state, and that is where training memory goes. LoRA also changes what you walk away with. A full fine-tune hands you a new model. LoRA hands you an adapter, a small file that sits beside the base weights, and one server can hold several of them at once and pick one per request. So post-training and serving meet here too, not just at the rollouts later in this chapter. Most people will hit this one rather than the rollouts, because it arrives with the method they are already using. None of this is specific to SFT. The same trick works under DPO and under the reinforcement learning methods, and it changes how many networks you have to hold. DPO needs a frozen copy of the model you started from. PPO and GRPO need a frozen reference. Under LoRA you already have one, because the base weights never moved, so switching the adapter off gives you the reference for nothing. The variants mostly differ in how the adapter itself is parameterised, which matters far less than the decision to use one at all. QLoRA is the exception worth knowing. It holds the frozen base at four bits and trains the adapter on top. That is part four's quantization turning up in the training stack, and it often decides whether a large model fits on hardware you own. When a model is badly behaved and the plan is reinforcement learning, the SFT data is the cheaper place to look first. Reinforcement learning cannot install a format or a tone the examples never showed. Eventually you want the model to prefer one answer over another where you cannot write the ideal response, only recognize it. That needs comparisons: this answer is better than that one. The old way was reinforcement learning from human feedback: train a reward model on the comparisons, then use RL to push the model toward high reward. It works, and it is a lot of machinery. **Direct preference optimization** removes most of it. You feed it triples: a prompt, the better answer, the worse one. It never builds the reward model and never runs the RL loop. Instead it optimizes one closed-form objective that reaches the same optimum the long route would, provided preferences follow the Bradley-Terry model. Same target, reached offline. That is not a promise of the same result on the data you actually have. In practice it is a classification loss against a frozen copy of the model you started with. That is why it became the default for preference work: no reward model to train, no rollouts to generate, and one loss function to tune. Comparison cannot teach everything. Sometimes what you want is not a style but a *correct answer* reached by a chain of reasoning. That is what reinforcement learning is for. **PPO**, proximal policy optimization, is the classical approach. Four networks are resident: the policy being trained, a reward model to score what it produces, a frozen reference to keep it from drifting, and a **critic**. The critic predicts how well a state should turn out, so the algorithm can tell whether an outcome beat expectations or fell short of them. *GRPO is the one that drops the critic. A critic roughly doubles what you have to train.* **GRPO** removes the critic. Instead of learning to predict the baseline, it samples a *group* of answers to the same prompt, scores them all, and uses the group's own mean as the baseline. Answers above it get reinforced, answers below it discouraged. **RLVR** makes this practical for reasoning. It stands for reinforcement learning *with* verifiable rewards. That is a different thing from reinforcement learning *from* human feedback, where people do the scoring. Here a program does it, so if the task has a checkable answer you need no reward model at all. A maths answer key, a unit test suite, a compiler. However, it can still be rigged: models special-case unit tests, exploit answer-extraction formats, and reach right answers by invalid routes the checker never sees. But then the problem is a hole in your checker, which is a program you can open and read. You cannot read a reward model the same way. GRPO with a verifier on top is the published recipe behind DeepSeek-R1 and the open reasoning models that followed it. Which one you use depends on what data you can get. Work down the list: **Start with SFT.** One network, no sampling, and the only method that works when you can write the answer down. If the problem is format, tone or domain knowledge, nothing further down this list will fix it better. **Add DPO when you can rank but not write.** A second network in memory, still no sampling, covering the subjective tasks that have no correct answer. Cheaper than anything below it on this list. **Reach for GRPO with a verifier when correctness is checkable.** Sampling is where the cost jumps, because generating G answers per prompt is a serving workload. It is not the only route to a reasoning gain. Plain SFT on reasoning traces from a stronger model has produced large gains on its own. There is also a live argument that RLVR sharpens what the base model already samples rather than adding capability. Its advantage over distillation is that it needs no stronger teacher. **Consider PPO last, and probably not at all.** Four resident networks against GRPO’s two. That two assumes the reward comes from a verifier. GRPO removes the critic, not the reward model, so with a learned reward model it is three. Either way you are paying for an advantage that only shows up when credit has to be assigned token by token across a long sequence. If you can neither check an answer nor rank it against another, reinforcement learning has no signal to work with at all. Your problem is the SFT data. **Reinforcement learning rollouts are inference.** A GRPO run samples a group of answers per prompt, eight or sixteen or thirty-two, for every prompt in every batch, for the whole run. That generation is ordinary decoding. It uses the model and the KV cache the way serving does, and it runs into the memory bandwidth limit from part one on the engines part five described. Most of your post-training time goes there. That means the arithmetic in this series applies to a training run. The KV cache still decides how big a batch you can hold, and rollouts have an advantage here: you generate G answers for every prompt, so the batch runs into the hundreds. Part one put the ridge on an H100 at 296 and said almost nobody sees it in practice. A rollout batch clears it, on a dense model at least. On a mixture of experts each expert only sees its own share, so the number that matters is smaller than the headline. The answers in a group also start from the same prompt, so they share a prefix exactly. Part two called this often the biggest single win available on the workloads that have one. Quantization is where the comparison stops. Part two’s doubling came from halving the *cache*. Halving the *weights* is a different thing. It hands back a fixed number of bytes once, and what that buys depends on how your budget is split. On part two’s numbers, 65.25 GB of weights become 32.6, freeing about 30 GiB, so a 71 GiB cache budget becomes roughly 101. That is 1.4×, not 2×. Two caveats. Those are serving numbers, borrowed to show the size of the effect rather than because a 120B policy and its optimizer state would fit on two cards. And gpt-oss-120b already ships its experts at four bits, so for this checkpoint the halving is hypothetical. Quantizing a rollout policy also costs something that quantizing a served model does not. The rollout policy is the model being trained. Quantize it and the version that generated the samples is no longer quite the version being updated. The objective works by comparing those two, so the comparison stops being trustworthy. In serving, quantization costs you quality, which is what part four is about and which you settle with your own evaluation. In a rollout you pay that and the broken comparison on top. *Weights and cache share one budget here, so halving the weights returns less than halving the cache did in part two.* The KV budget is tighter than part two’s as well, because in a colocated run the cache only gets what is left after weights, gradients, optimizer state and activations. Training and serving hit the same limit. Fix the SFT data before reaching for anything with “reinforcement” in the name. That is cheaper, faster, and sometimes, more effective. It is also usually where the real problem was. If you are running rollouts at any scale, profile the generation separately from the training step. If most of your time is in generation, and it very often is, the seven preceding parts of this series are about your training job too. Post-training and serving get treated as two disciplines with two vocabularies. SFT and DPO have nothing to do with paged attention or tensor parallelism. But the moment a method generates its own samples to learn from, it is doing inference, and everything the rest of this series says about inference applies to it. So it is worth knowing whether yours does. *The whole chapter on one image. What decides the method is your use case and what you have to check the work with.* If you have run any of these, I would like to hear which one and how it went. Please clap or comment and follow me. The rest of this series is about serving the model rather than training it, and it starts with [part one](https://medium.com/towards-artificial-intelligence/start-here-the-words-everyone-uses-about-llm-inference-f4f1edc4a1b2). [SFT, RL and DPO: The Other Stack](https://pub.towardsai.net/sft-rl-and-dpo-the-other-stack-0ab7026d528e) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.