14 min read · AI Research · LLM Training
I want to tell you about the moment I stopped thinking about AI as a prediction machine.
It happened when I was reading through the DeepSeek-R1 paper at an ungodly hour — the kind of reading you do when something genuinely bothers you and you can’t sleep until you understand it. I had been following the usual discourse: bigger models, more data, better benchmarks. The standard story. And then I hit the section where the researchers described something they hadn’t engineered, hadn’t planned for, hadn’t trained explicitly.
The model had started talking to itself.
Not hallucinating. Not repeating. It was backtracking — catching its own mistakes mid-reasoning, reconsidering its approach, trying again. The researchers called it an “aha moment.” I called it the most interesting thing I had read in months. And the technique that caused it wasn’t a new architecture or a trillion more tokens of training data.
It was a change in how the model was rewarded.
That’s what RLVR is. And once you understand it, you can’t unsee it — in DeepSeek-R1, in o3, in GPT-6 Astra, in basically every frontier model that matters in 2026.
Let me back up.
For years, the dominant approach to making language models useful was something called Reinforcement Learning from Human Feedback — RLHF. The idea seems sensible on the surface: you take a pretrained model, generate some responses, ask human raters to rank them from best to worst, train a reward model on those rankings, and then use reinforcement learning to push the main model toward responses the reward model likes.
And it works. RLHF is why ChatGPT felt so much better than raw GPT-3. It’s why models stopped generating offensive content unprompted. It genuinely made AI more usable.
But I want to be honest about something that took me a while to fully sit with: RLHF is not teaching models to be correct. It’s teaching them to be liked.
Think about what that actually means. A human rater sees two responses and picks the one that seems better. But human raters get tired. They have biases. They prefer responses that sound confident, that are well-formatted, that feel thorough — regardless of whether the underlying reasoning is sound. A long, fluent, beautifully structured wrong answer will beat a short, clunky right one almost every time.
So you end up with models that are extraordinarily good at sounding correct. At performing competence. At generating the shape of a good answer without necessarily having the substance of one.
I remember testing this myself — asking models to solve problems I already knew the answers to, specifically in mathematics. The responses were articulate, well-organized, completely wrong, and utterly confident. The model had learned to produce the genre of a correct mathematical answer. It had not learned to do mathematics.
This isn’t a criticism of the researchers who built RLHF — it was a genuine step forward and a necessary one. But it’s a ceiling. And for a while, nobody had a clean idea of how to break through it.
The insight behind RLVR is almost embarrassingly simple once you hear it.
Instead of asking a human “is this response good?”, you ask a function “is this answer correct?”
That’s it. That’s the core of Reinforcement Learning with Verifiable Rewards.
Rather than training a reward model on human preferences — which are subjective, inconsistent, and expensive to collect — you define a verifier: a deterministic function that can check whether the model’s output is actually right. Reward = 1 if correct. Reward = 0 if wrong. No ambiguity. No human in the loop. No possibility of a fluent wrong answer sneaking through.
The verifier doesn’t care how the model expressed the answer. It doesn’t care if the response was well-formatted or confidently worded. It checks the answer against ground truth and returns a binary signal. Pass or fail.
This works beautifully in two domains that have clear, checkable answers:
Mathematics — Did the final numerical answer match the reference? Run a symbolic checker. You don’t need a human to tell you whether 42 is the right answer to a specific integral.
Code — Did the code pass the test suite? Run the tests. You don’t need a human to evaluate whether a function correctly sorts a list — the tests either pass or they don’t.
In both cases, you suddenly have an unlimited supply of training signal that is cheap, fast, consistent, and impossible to fool with confident-sounding nonsense. A model cannot produce a fluent-sounding but incorrect answer and get rewarded. Either the code runs or it doesn’t. Either the math is right or it isn’t.
The RLVR training loop looks like this: the model receives a prompt, generates multiple candidate responses, each response passes through the verifier, the verifier returns a reward signal, and the model’s policy is updated to make correct responses more likely. Repeat, at scale, for millions of problems.
What emerges from that loop — and this is the part that surprised even the researchers — is something that looks a lot like genuine reasoning.
Before we get to what RLVR produced in practice, it’s worth understanding the algorithms that power it — because the choice of algorithm turns out to matter enormously.
Three algorithms define the modern post-training landscape. They’re worth understanding as a progression rather than alternatives.
Proximal Policy Optimization was the original workhorse of RLHF-based LLM training. It’s what OpenAI used to build InstructGPT — the model that became the basis for ChatGPT.
PPO works like this: the model generates a response, a reward model scores it, and then PPO updates the policy — but with a constraint. It can’t stray too far from the original model in a single update. The “proximal” in the name refers to that guardrail: stay close to what you already know, update cautiously, don’t collapse into reward-hacking behaviour.
The problem is computational. PPO needs to run a value model — a separate neural network that estimates the expected future reward from any given state — alongside the policy model at training time. For a large language model, that means running two massive models simultaneously during training. The memory requirements are brutal, the engineering complexity is high, and stability is notoriously difficult to maintain.
PPO works. But it’s expensive and finicky — not ideal for the scale at which modern labs need to operate.
Direct Preference Optimization arrived in 2023 and caused genuine excitement, because it seemed to solve the PPO complexity problem entirely.
DPO’s key insight is that the reward model and the RL training loop are actually redundant. If you have human preference data — pairs of responses where one is preferred over the other — you can derive the optimal policy directly from that data, without ever training a reward model or running an RL loop.
The training feels like a standard fine-tune: here are some (prompt, chosen response, rejected response) triples; optimize the model to prefer the chosen one. No environment, no sampling, no advantage estimation. Dramatically simpler, more stable, and faster than PPO.
The catch — and this is the critical limitation — is that DPO still optimises for human preferences. It’s a cleaner, cheaper way to do the same thing RLHF was doing. Which means it inherits the same fundamental ceiling: you’re optimising for what humans prefer, not for what is correct.
DPO is excellent for alignment, tone, and following instructions. It is not a path to genuine mathematical reasoning.
Group Relative Policy Optimization is DeepSeek’s contribution to this progression, and it’s the reason DeepSeek-R1 was possible.
GRPO solves the PPO scaling problem in a clever way: instead of needing a separate value model to estimate expected future reward, it estimates the advantage of a response relative to a group of other responses for the same prompt.
Here’s how it works in practice. For a given math problem, the model generates — say — eight different candidate solutions. The verifier checks each one. Some are correct, some aren’t. GRPO then computes the relative advantage: if your response is correct and most of the group isn’t, you get a strong positive signal. If your response is wrong and most of the group got it right, you get a negative signal. The group itself is the baseline — no separate critic network required.
This eliminates the value model entirely. You get RL-quality training signal with something approaching DPO-level computational simplicity. And when you pair GRPO with RLVR’s verifiable rewards — where the verification is cheap, fast, and deterministic — you have a training paradigm that can scale.
Here’s a clean way to see how the three compare:
GRPO didn’t just make RLVR more efficient. It made it possible to run RLVR at the scale required to see something genuinely new emerge.
I’ll keep this section tight because the result speaks for itself.
DeepSeek-R1 used GRPO-powered RLVR at scale on mathematical and reasoning tasks. The researchers did not explicitly train the model to reason step-by-step, to check its work, or to backtrack when it made mistakes. They trained it to get the right answer, and let the reward signal figure out the rest.
What emerged was spontaneous chain-of-thought reasoning. Self-correction. The model pausing mid-response, flagging an error in its own logic, and trying a different approach. Behaviours that nobody programmed in — that the training signal discovered as useful strategies for getting verifiable rewards.
The researchers literally called it an “aha moment” in the paper, which is not standard academic language. It was their way of saying: we didn’t expect this, and it surprised us too.
And DeepSeek used only about 5% of their overall compute in post-training — roughly 147,000 GPU hours for the RL stage, compared to 2.8 million GPU hours for pretraining the underlying base model. The intelligence gains were disproportionate to the compute investment. That ratio is what made the AI world pay attention.
Once DeepSeek proved the concept, every major lab moved fast.
OpenAI’s o3 combined RLVR-style training with test-time compute — letting the model “think longer” during inference on hard problems — and the results on benchmarks like ARC-AGI and GPQA Diamond shocked even the researchers who designed those benchmarks. Gemini 3 Deep Think followed a similar path and currently leads the ARC-AGI-1 leaderboard at around 96%.
The key point isn’t the specific scores. It’s the mechanism: these models got dramatically smarter not because they were trained on more data or had more parameters, but because the training signal itself got better. RLVR told them to be correct, not just liked. And they responded by learning to actually reason.
This is where the story gets genuinely interesting — and genuinely recent.
GPT-6 Astra (OpenAI’s internal codename was “Bel”) launched to partners on 3 September 2026, making it one of the freshest data points we have on where RLVR is heading. And the training details that have emerged paint a fascinating picture.
Pre-training ended around July 2026. Then something notable happened: the RL training phase was d on 14 August and resumed on 28 August. OpenAI hasn’t explained why publicly, but that two-week in the middle of RL training on what is likely the most expensive model ever built is a detail worth sitting with. RL training at frontier scale is not a clean, predictable process. Something needed to be fixed, adjusted, or reconsidered. The fact that this happened — and that they resumed and shipped — tells you something about both the difficulty and the commitment.
What Astra can do, according to OpenAI’s own demonstrations, goes well beyond what RLVR was achieving even six months ago. In one demonstration, 16 AI agents divide a research-level mathematics problem into subproblems, coordinate their work across agents, and assemble a proposed proof. In another, Astra navigates desktop software — creating and editing work across applications — in what observers described as a “superhuman, very fast kind of way.”
And Sam Altman’s framing of Astra is worth quoting directly. He called it “the first model where the model actually invents new things in a way that matters.” He said it can take an experimental idea, implement it inside OpenAI’s codebase, run the experiment, and return results — or take a paper and perform work that previously occupied a human researcher for a week.
That is not the description of a model that learned to sound like it reasons. That is the description of a model that has been trained — through RL at enormous scale — to actually do things in the world and get them right.
Altman had actually telegraphed this two years earlier. In February 2025, he said: “GPT-3 and GPT-4 are pre-training paradigms. GPT-5 and GPT-6 will utilize reinforcement learning and will be like discovering new science, such as new algorithms, physics, and biology.”
He was right. And the technique that made it possible traces directly back to the same insight that powered DeepSeek-R1: reward the model for being correct, not for being liked, and give it enough attempts to figure out how.
Here is the part of RLVR that keeps researchers up at night — and honestly, it’s the part I find most intellectually interesting.
RLVR requires a verifiable reward signal. And that requirement, which is also its greatest strength, is also its most serious limitation.
Mathematics has right answers. Code either passes tests or it doesn’t. But what about:
Most real-world tasks don’t have a clean, checkable answer. Extending RLVR to these domains requires either constructing clever proxy verifiers — which risks reward hacking, where the model learns to satisfy the verifier without actually solving the problem — or rethinking what “verifiable” even means.
The reward hacking failure mode is genuinely concerning. A model optimising against an imperfect verifier will find the holes. It will discover ways to score the maximum reward without doing the underlying thing the reward was meant to incentivise. This is not hypothetical — it has been observed in practice, including in mathematical domains where models learned to format their final answers in ways that tricked symbolic checkers.
Current approaches to extending RLVR include process reward models, which reward correct intermediate steps rather than just final answers; self-verifiable rewards, where the model is trained on tasks that generate their own verification signal through interaction; and composite reward functions that combine multiple signals to make gaming harder.
None of these are fully solved. The field is moving fast, but the gap between “works well on math and code” and “works well on open-ended real-world reasoning” remains large.
Let me try to say something a bit bigger than a technical summary here, because I think the implications of RLVR go beyond the benchmarks.
For most of the history of modern AI, the training signal came from humans. Human-written text to pretrain on. Human preferences to align to. Human feedback to fine-tune against. The model was, fundamentally, an approximation of human judgment at scale.
RLVR breaks that dependency — partially, in specific domains. When you can verify correctness without a human in the loop, you can train at a speed, scale, and consistency that human feedback can never match. You can generate millions of training examples overnight. You can run them through a verifier in seconds. You can iterate in ways that would take years if you needed human raters for every step.
And the models that emerge from this process don’t just perform better on benchmarks. They seem to develop qualitatively different behaviours — the spontaneous reasoning, the self-correction, the ability to decompose hard problems — that weren’t explicitly programmed in.
The deeper implication, which I don’t think the field has fully reckoned with, is this: if you can train a model to be correct in a domain by giving it unlimited verifiable practice at scale, you don’t need that domain to have a long history of human-annotated training data. You need a good verifier and enough compute. That changes the economics of capability development in ways that are still working themselves out.
Sam Altman described GPT-6 Astra as a model that “invents new things in a way that matters.” If that claim holds up under scrutiny, it represents something genuinely new: a system trained primarily through verifiable reward signals that has developed the capacity to generate novel correct outputs — not just retrieve or recombine existing ones.
Whether that constitutes “invention” in any meaningful sense is a philosophical question worth taking seriously. What’s not in question is the mechanism that got us here: reward the model for being right, give it enough tries to figure out how, and something interesting emerges.
When I started digging into RLVR properly, I thought I was learning about a training technique. A method. A tool in a toolkit.
What I came to realise is that it represents a shift in what we think AI training is for. The RLHF era was about making AI more agreeable, more aligned with human preferences, more polished in its outputs. That was genuinely valuable. But it was also, in a sense, training AI to be a better mirror of what we already think.
RLVR — at least in domains where it works — is training AI to find things that are true independent of what we prefer. To be correct, not just acceptable. To reason toward the right answer rather than the liked one.
That distinction matters more than it might sound. As these systems take on harder problems — in mathematics, in science, eventually in domains we haven’t figured out how to verify yet — the question of whether they’re optimising for human approval or for actual correctness becomes one of the most important questions in the field.
I don’t have a clean conclusion here. The technique is young, the domains are limited, and the hardest problems are still unsolved. But something changed when DeepSeek’s model started catching its own mistakes in the middle of a reasoning chain — something that nobody programmed in, that emerged purely from the pressure of having to be right.
I find that genuinely remarkable. And I think we’re only at the beginning of understanding what it means.
If something here clicked differently than you expected — or if you think I’ve got something wrong — tell me in the comments. The best understanding comes from being challenged.
Tags: LLM Training · RLVR · Reinforcement Learning · DeepSeek R1 · GPT-6 · AI Research · Machine Learning
RLVR: From Human Feedback to Verifiable Truth: How Modern LLMs Actually Learn to Reason was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.