{"slug": "how-llms-learned-to-reason-sft-rlhf-rlvr", "title": "How LLMs Learned to Reason: SFT --> RLHF --> RLVR", "summary": "A developer explains the evolution of large language model training from supervised fine-tuning to reinforcement learning from human feedback and reinforcement learning with verifiable rewards, noting that major models like GPT-4.5 and DeepSeek-V3 were among the last built on the pretrain-then-instruct recipe. The post highlights the shift toward training models to reason before answering, using techniques like PPO and GRPO to overcome the limitations of SFT.", "body_md": "GPT-4.5, DeepSeek-V3, and Claude 3.5 Sonnet share something that has nothing to do with benchmark scores: they were the last major models built entirely on the \"pretrain, then instruct-tune\" recipe. All internal computation was done in one forward pass per token, with no backtracking, verification or revision mechanisms in place.\n\nBy late 2024, these labs had run into the same big wall: scaling pretraining data and compute had reached a point of saturation. The naive recipe of more tokens, bigger model, more GPU-compute no longer bought equivalent capability gains, particularly on multi-step reasoning tasks. Good output quality became the bottleneck before parameter count did.\n\nWhat came after wasn't a bigger version of the same thing. It was a different training paradigm applied *on top of* the existing ones. The field pivoted to training models to think before they answer.\n\nThink of a Roomba cleaning a house. The **environment** is the house; the **state** is what the Roomba currently senses (position, dirt map, obstacles); an **action** is a movement decision (turn, advance, suck); a **trajectory** is one full cleaning run; and the **reward** is some measure of how much dirt got picked up, dispensed along the way or tallied at the end.\n\nMap this onto an LLM generating text: the **state** is the prompt plus every token generated so far; an **action** is the next token (or, at a coarser grain, the next reasoning step); a **trajectory** is the full generated sequence; and the **reward** is a scalar signal applied to that sequence, either at the very end or, in some setups, at intermediate points.\n\n**Pretraining → SFT → RLHF / RLVR**\n\nPretraining gives the model raw capability and world knowledge from next-token prediction over a huge corpus. SFT and the RL stages that follow are about *shaping* that capability toward useful, correct, well-formed behaviour. The rest of this post is a section-by-section zoom into each stage after pretraining.\n\nSupervised fine-tuning takes a pretrained base model and fine-tunes it on curated (prompt, response) pairs. This is typically written or heavily edited by human annotators to model the response style and quality. It's imitation learning: the model isn't exploring or being scored on outcomes, it's just learning to match a distribution of demonstrated behaviour.\n\nThis works well for instruction-following, tone, and formatting.\n\nIt eventually plateaus for two structural reasons:\n\nThis is the core limitation RL-based methods are built to address: give the model a way to try multiple candidates and get some sort of a signal on which one was actually better.\n\nThe classic RLHF pipeline has 3 stages: an SFT model, a reward model trained on human preference comparisons, and a gradient step that optimises the SFT model against that reward model.\n\n**How it works**: \n\n1) Humans are shown several candidate responses to the same prompt and rank them by preference. \n\n2) That ranking data trains a separate reward model to predict, for any given response, how a human would score it. \n\n3) Once the reward model is trained, it stands in for the human and the policy (LLM) is optimised directly, using [PPO (Proximal Policy Optimization)](https://arxiv.org/pdf/1707.06347)\n\n4) The labelling process effectively automates itself after the initial ranking data is collected.\n\nI shall digress here a bit to mention PPO is *no longer* the default anymore. DeepSeek's papers (DeepSeekMath, DeepSeek-V3, DeepSeek-R1) use [**GRPO** (Group Relative Policy Optimization)](https://arxiv.org/pdf/2402.03300) instead.\n\nWorth a separate discussion on its own, so we'll save the full internals for a separate blog and just sum it up here below. (Note: Policy <--> LLMs in this context and topic)\n\n**PPO**: The standard policy-gradient method. Generate a response, score it, and update the policy in the direction that increases reward while a clipping term keeps each update close to the old policy to not destabilise training. \n\nDoing this well requires a separate critic network (~same size as the policy) to estimate expected reward at each step, which is expensive since LLM reward is typically only assigned at the end of a sequence.\n\n**GRPO**: Same clipped policy-gradient, but drops the critic network. For each prompt, it samples a group of candidate outputs from the current policy, scores each, and uses the group's own mean/variance as the advantage baseline directly which required no learned value estimates. A response better than the group average gets reinforced, one worse than average gets suppressed. Cheaper and simpler.\n\nRegardless of PPO vs. GRPO, the more important issue is *what the reward model is trained to predict in this step*: human preference, not correctness. This acts as a good proxy for tone, helpfulness, and safety, but it's a poor proxy for reasoning tasks specifically. A human rater judges plausibility, confidence and structure, often not the actual correctness of the output. \n\nThis leads us to the limitation where a confidently wrong answer, well formatted and persuasively argued, can out-score a correct but messier one.\n\nRLVR comes after/along with the learned reward model with something more fitting for reasoning tasks: a programmatic and deterministic checker for correctness, with no learned approximation in the loop.\n\n**How it Works**:\n\n1) Code: The model's output is compiled to run against unit tests similar to a LeetCode compiler. Pass/Fail quantifies the reward.\n\n2) Math: The final answer is checked against a known ground truth (calculated output), with normalisation (`1/2` == `0.5`).\n\n3 Structured/Rule-governed tasks (grammar correction, formatting compliance): Rule-based checker validates the output against the fixed specifications.\n\n**Why this unlocks reasoning gains that RLHF alone didn't reliably produce**:\n\n1) No reward hacking via sycophancy. A learned reward model can be gamed by outputs that are *seemingly* good to the reward model's learned heuristics (confident tone, agreeable phrasing). A compiler has no aesthetic preferences. It either passes or fails.\n\n2) The reward signal scales with problem difficulty. A model gets no partial credit from a unit-test suite for writing confidently and the result is rather deterministic, it either produces working code or it doesn't, regardless of how hard the underlying problem was.\n\nDeepSeek released three closely related artifacts in January 2025 that, taken together, answer three separate questions:\n\nR1-Zero was trained directly from a DeepSeek-V3-Base checkpoint using large-scale RLVR (via GRPO). This had no supervised fine-tuning step beforehand.\n\nThe outcome being reasoning benchmark accuracies climbed substantially, and the model spontaneously developed longer chains of thought and self-verification-like behaviour. None of which were part of the reward function.\n\nBut the model also developed serious presentation problems: reasoning chains would mix languages mid-thought (English and Chinese within the same reasoning chains), general readability was poor with the raw output being difficult to follow even when the final answer was correct.\n\nThis is the direct consequence of optimising purely for a verifiable outcome reward with no constraint pushing the intermediate reasoning. The reward function never mentioned the **chain** being legible, only the final answer mattered.\n\n*The `a-ha` moment as highlighted for R1-zero as in their famous paper. Source: [DeepSeek Paper](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf)*\n\nThe fix to the above section was to reintroduce a small amount of supervision before the RL stage (\"cold start SFT\"): a modest set of curated, high-quality chain-of-thought examples used to fine-tune the base model via SFT *before* running RLVR. \n\nThis cold-start data was specifically curated for readability and consistent language use, giving the model a coherent starting distribution to explore from, rather than leaving it to discover legible reasoning formats on its own.\n\nSubsequently, the pipeline proceeded through further RL stages (including a language-consistency reward), followed by rejection sampling and additional SFT on the resulting high-quality outputs, before a final RL pass.\n\nThe result, per DeepSeek's reported benchmarks, is a model that performs on par with OpenAI's o1 on math, code, and STEM reasoning benchmarks, while producing coherent, single-language, readable reasoning traces. A direct nudge away from R1-zero.\n\n*Source: [Cohere's Thoughtology Paper](https://arxiv.org/abs/2504.07128)*\n\nThe third piece to this case study answers the question: does the reasoning capability in a 671B-parameter model require RL to reproduce reasoning in a smaller model, or can it just be copied?\n\nDeepSeek generated a large set of reasoning traces from the full R1 model and used them purely as SFT data to fine-tune smaller dense models (Qwen2.5 and Llama3 based) ranging from 1.5B to 70B parameters. No RL stage was run on the smaller models at all. Just imitation. (Note: We are not talking about Logits-based distillation here, this was purely used as SFT)\n\nThe result is genuinely counterintuitive given everything above in R1-zero covered SFT's limitations: distillation from R1 outperformed running RL from scratch directly on the smaller base models.\n\nDeepSeek compared distilling into Qwen2.5-32B against applying the same large-scale RLVR recipe directly to a 32B base model, and the distilled version won.\n\nA few concrete distilled-model numbers from DeepSeek's released benchmarks (AIME 2024 pass@1):\n\n*Source: [DeepSeek’s release paper](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf)*\n\nThe exploration concurred, good reasoning strategies via RLVR seem to require the extra capacity of a large base model. Once those strategies exist as demonstrated behaviour, a much smaller model can absorb them through plain imitation.\n\nWe went from single-pass, non-reasoning flagships to models trained with verifiable rewards to actually reason: SFT gave a plateauing imitation baseline, RLHF added preference optimisation but mismatched on correctness, and RLVR fixed that mismatch with checkable outcomes. DeepSeek's R1 lineage showed what this looks like in practice. R1-Zero's raw, undirected RL reasoning; R1's cold-start fix; and R1-Distill's counterintuitive proof that those reasoning patterns transfer via simple imitation, no RL required downstream.\n\nOne side effect along the way: as RL training progressed, chain-of-thought length grew on its own and tracked with accuracy on harder problems, reasoning longer turned out to be a strategy the model discovered, not one it was told to use.", "url": "https://wpnews.pro/news/how-llms-learned-to-reason-sft-rlhf-rlvr", "canonical_source": "https://dev.to/cyprus09/how-llms-learned-to-reason-sft-rlhf-rlvr-1ldh", "published_at": "2026-09-09 00:09:45+00:00", "updated_at": "2026-09-09 00:15:39.446974+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-infrastructure"], "entities": ["OpenAI", "DeepSeek", "Anthropic", "GPT-4.5", "DeepSeek-V3", "Claude 3.5 Sonnet", "PPO", "GRPO"], "alternates": {"html": "https://wpnews.pro/news/how-llms-learned-to-reason-sft-rlhf-rlvr", "markdown": "https://wpnews.pro/news/how-llms-learned-to-reason-sft-rlhf-rlvr.md", "text": "https://wpnews.pro/news/how-llms-learned-to-reason-sft-rlhf-rlvr.txt", "jsonld": "https://wpnews.pro/news/how-llms-learned-to-reason-sft-rlhf-rlvr.jsonld"}}