How LLMs Learned to Reason: SFT --> RLHF --> RLVR

A developer explains the evolution of large language model training from supervised fine-tuning to reinforcement learning from human feedback and reinforcement learning with verifiable rewards, noting that major models like GPT-4.5 and DeepSeek-V3 were among the last built on the pretrain-then-instruct recipe. The post highlights the shift toward training models to reason before answering, using techniques like PPO and GRPO to overcome the limitations of SFT.

GPT-4.5, DeepSeek-V3, and Claude 3.5 Sonnet share something that has nothing to do with benchmark scores: they were the last major models built entirely on the "pretrain, then instruct-tune" recipe. All internal computation was done in one forward pass per token, with no backtracking, verification or revision mechanisms in place. By late 2024, these labs had run into the same big wall: scaling pretraining data and compute had reached a point of saturation. The naive recipe of more tokens, bigger model, more GPU-compute no longer bought equivalent capability gains, particularly on multi-step reasoning tasks. Good output quality became the bottleneck before parameter count did. What came after wasn't a bigger version of the same thing. It was a different training paradigm applied on top of the existing ones. The field pivoted to training models to think before they answer. Think of a Roomba cleaning a house. The environment is the house; the state is what the Roomba currently senses position, dirt map, obstacles ; an action is a movement decision turn, advance, suck ; a trajectory is one full cleaning run; and the reward is some measure of how much dirt got picked up, dispensed along the way or tallied at the end. Map this onto an LLM generating text: the state is the prompt plus every token generated so far; an action is the next token or, at a coarser grain, the next reasoning step ; a trajectory is the full generated sequence; and the reward is a scalar signal applied to that sequence, either at the very end or, in some setups, at intermediate points. Pretraining → SFT → RLHF / RLVR Pretraining gives the model raw capability and world knowledge from next-token prediction over a huge corpus. SFT and the RL stages that follow are about shaping that capability toward useful, correct, well-formed behaviour. The rest of this post is a section-by-section zoom into each stage after pretraining. Supervised fine-tuning takes a pretrained base model and fine-tunes it on curated prompt, response pairs. This is typically written or heavily edited by human annotators to model the response style and quality. It's imitation learning: the model isn't exploring or being scored on outcomes, it's just learning to match a distribution of demonstrated behaviour. This works well for instruction-following, tone, and formatting. It eventually plateaus for two structural reasons: This is the core limitation RL-based methods are built to address: give the model a way to try multiple candidates and get some sort of a signal on which one was actually better. The classic RLHF pipeline has 3 stages: an SFT model, a reward model trained on human preference comparisons, and a gradient step that optimises the SFT model against that reward model. How it works : 1 Humans are shown several candidate responses to the same prompt and rank them by preference. 2 That ranking data trains a separate reward model to predict, for any given response, how a human would score it. 3 Once the reward model is trained, it stands in for the human and the policy LLM is optimised directly, using PPO Proximal Policy Optimization https://arxiv.org/pdf/1707.06347 4 The labelling process effectively automates itself after the initial ranking data is collected. I shall digress here a bit to mention PPO is no longer the default anymore. DeepSeek's papers DeepSeekMath, DeepSeek-V3, DeepSeek-R1 use GRPO Group Relative Policy Optimization https://arxiv.org/pdf/2402.03300 instead. Worth a separate discussion on its own, so we'll save the full internals for a separate blog and just sum it up here below. Note: Policy <-- LLMs in this context and topic PPO : The standard policy-gradient method. Generate a response, score it, and update the policy in the direction that increases reward while a clipping term keeps each update close to the old policy to not destabilise training. Doing this well requires a separate critic network ~same size as the policy to estimate expected reward at each step, which is expensive since LLM reward is typically only assigned at the end of a sequence. GRPO : Same clipped policy-gradient, but drops the critic network. For each prompt, it samples a group of candidate outputs from the current policy, scores each, and uses the group's own mean/variance as the advantage baseline directly which required no learned value estimates. A response better than the group average gets reinforced, one worse than average gets suppressed. Cheaper and simpler. Regardless of PPO vs. GRPO, the more important issue is what the reward model is trained to predict in this step : human preference, not correctness. This acts as a good proxy for tone, helpfulness, and safety, but it's a poor proxy for reasoning tasks specifically. A human rater judges plausibility, confidence and structure, often not the actual correctness of the output. This leads us to the limitation where a confidently wrong answer, well formatted and persuasively argued, can out-score a correct but messier one. RLVR comes after/along with the learned reward model with something more fitting for reasoning tasks: a programmatic and deterministic checker for correctness, with no learned approximation in the loop. How it Works : 1 Code: The model's output is compiled to run against unit tests similar to a LeetCode compiler. Pass/Fail quantifies the reward. 2 Math: The final answer is checked against a known ground truth calculated output , with normalisation 1/2 == 0.5 . 3 Structured/Rule-governed tasks grammar correction, formatting compliance : Rule-based checker validates the output against the fixed specifications. Why this unlocks reasoning gains that RLHF alone didn't reliably produce : 1 No reward hacking via sycophancy. A learned reward model can be gamed by outputs that are seemingly good to the reward model's learned heuristics confident tone, agreeable phrasing . A compiler has no aesthetic preferences. It either passes or fails. 2 The reward signal scales with problem difficulty. A model gets no partial credit from a unit-test suite for writing confidently and the result is rather deterministic, it either produces working code or it doesn't, regardless of how hard the underlying problem was. DeepSeek released three closely related artifacts in January 2025 that, taken together, answer three separate questions: R1-Zero was trained directly from a DeepSeek-V3-Base checkpoint using large-scale RLVR via GRPO . This had no supervised fine-tuning step beforehand. The outcome being reasoning benchmark accuracies climbed substantially, and the model spontaneously developed longer chains of thought and self-verification-like behaviour. None of which were part of the reward function. But the model also developed serious presentation problems: reasoning chains would mix languages mid-thought English and Chinese within the same reasoning chains , general readability was poor with the raw output being difficult to follow even when the final answer was correct. This is the direct consequence of optimising purely for a verifiable outcome reward with no constraint pushing the intermediate reasoning. The reward function never mentioned the chain being legible, only the final answer mattered. The a-ha moment as highlighted for R1-zero as in their famous paper. Source: DeepSeek Paper https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek R1.pdf The fix to the above section was to reintroduce a small amount of supervision before the RL stage "cold start SFT" : a modest set of curated, high-quality chain-of-thought examples used to fine-tune the base model via SFT before running RLVR. This cold-start data was specifically curated for readability and consistent language use, giving the model a coherent starting distribution to explore from, rather than leaving it to discover legible reasoning formats on its own. Subsequently, the pipeline proceeded through further RL stages including a language-consistency reward , followed by rejection sampling and additional SFT on the resulting high-quality outputs, before a final RL pass. The result, per DeepSeek's reported benchmarks, is a model that performs on par with OpenAI's o1 on math, code, and STEM reasoning benchmarks, while producing coherent, single-language, readable reasoning traces. A direct nudge away from R1-zero. Source: Cohere's Thoughtology Paper https://arxiv.org/abs/2504.07128 The third piece to this case study answers the question: does the reasoning capability in a 671B-parameter model require RL to reproduce reasoning in a smaller model, or can it just be copied? DeepSeek generated a large set of reasoning traces from the full R1 model and used them purely as SFT data to fine-tune smaller dense models Qwen2.5 and Llama3 based ranging from 1.5B to 70B parameters. No RL stage was run on the smaller models at all. Just imitation. Note: We are not talking about Logits-based distillation here, this was purely used as SFT The result is genuinely counterintuitive given everything above in R1-zero covered SFT's limitations: distillation from R1 outperformed running RL from scratch directly on the smaller base models. DeepSeek compared distilling into Qwen2.5-32B against applying the same large-scale RLVR recipe directly to a 32B base model, and the distilled version won. A few concrete distilled-model numbers from DeepSeek's released benchmarks AIME 2024 pass@1 : Source: DeepSeek’s release paper https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek R1.pdf The exploration concurred, good reasoning strategies via RLVR seem to require the extra capacity of a large base model. Once those strategies exist as demonstrated behaviour, a much smaller model can absorb them through plain imitation. We went from single-pass, non-reasoning flagships to models trained with verifiable rewards to actually reason: SFT gave a plateauing imitation baseline, RLHF added preference optimisation but mismatched on correctness, and RLVR fixed that mismatch with checkable outcomes. DeepSeek's R1 lineage showed what this looks like in practice. R1-Zero's raw, undirected RL reasoning; R1's cold-start fix; and R1-Distill's counterintuitive proof that those reasoning patterns transfer via simple imitation, no RL required downstream. One side effect along the way: as RL training progressed, chain-of-thought length grew on its own and tracked with accuracy on harder problems, reasoning longer turned out to be a strategy the model discovered, not one it was told to use.