LLMs are (still) mostly powered by imitative learning, not RL

LLMs derive most of their capabilities from imitative learning (pretraining and supervised fine-tuning), not from reinforcement learning from verifiable rewards (RLVR), according to a LessWrong analysis. The author argues that RLVR accounts for a tiny fraction of the information content and weight changes in trained models, and that chain-of-thought legibility remains high because imitative learning dominates. This matters for understanding chain-of-thought interpretability, LLM capabilities, and alignment, though RLVR still contributes meaningfully to performance.

Reinforcement learning from verifiable rewards RLVR is the hot new thing in LLM training. It’s so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture. Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of: If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM’s capabilities. And my claim is that it’s way more 1 than 2 . I'll start in §1 with some relevant evidence, and then in §2 I’ll circle back to operationalizing exactly what I’m claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment. Note that I am not arguing that RLVR does not importantly contribute to LLM capabilities. That would be absurd Of course it does Companies use RLVR because it works, and I expect them to continue doing so more and more. Again, the things I’m actually claiming are in §2–§3. The fraction of total LLM training compute that goes towards RLVR is high I vaguely recall hearing 50% . But that’s different from how to divvy up the credit that pretraining/SFT versus RLVR can take for LLM capabilities. Why is that different? See: Based on those posts, if RLVR is 50% of the training compute, then it might only be 0.01% of the information content imparted by training. And unless the learning rates are drastically different, it would also be 0.01% of changes to the weights. People emphasize how frontier model CoTs can be pretty weird, and that’s true and important, but if you step back a bit, in the grand scheme of things, CoTs are still very obviously strongly influenced by imitative learning from human-created and hence legible text. See for example, I prefer to start from the opposite side. What would a really inscrutable CoT look like? It would look like random tokens: “… extracurricular boolean ger застос autism relevance directamenteдвэр QUICKchem …” CoTs sure don’t look like that So on a spectrum from “totally legible” to “looks like random tokens”, we are way closer to the “totally legible” side, as of today. What does that imply about how LLMs get their superpowers? Well, imitative learning by itself will necessarily create legible CoTs; whereas in RL, the CoT is generally irrelevant to the reward, so there’s no force keeping it legible, and thus we should expect it to drift into an ever less legible and more “optimized” 2 direction. Thus, the fact that CoTs are still pretty legible is some evidence that RLVR today has not had that much influence on LLMs, as compared to imitative learning. See “Most Algorithmic Progress is Data Progress” https://www.beren.io/2025-08-02-Most-Algorithmic-Progress-is-Data-Progress/ Beren Millidge 2025 , plus further discussion in my post “The nature of LLM algorithmic progress” §1.4 https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK/the-nature-of-llm-algorithmic-progress 1 4 Data related improvements . And LLM companies today continue to spend money on even more training data for imitative learning especially expert reasoning traces for SFT, I think . 3 https://www.lesswrong.com/feed.xml fnugrvgvqxrnf The authors found that when an RLVR’d model could solve a problem, the equivalent non-RLVR’d model could generally solve the same problem, maybe not on the first try, but in one of the first 1000 tries or whatever. The authors found that an “iterative sampling algorithm” could boost a non-RLVR’d model to be comparable to the corresponding RLVR’d model. The authors found that you can take a non-RLVR’d model, randomly perturb the weights, and ensemble the results by majority vote, and you can get results comparable to an RLVR’d model. Caveat for all of these: I don’t think we should over-anchor on any of these, mainly because they’re all now outdated, and based on open models rather than bleeding-edge LLMs with bleeding-edge RLVR best practices. Also, there’s another paper “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models” Liu et al. 2025, arxiv which claims “that RL-trained models consistently outperform base models … including scenarios where base models fail entirely regardless of the number of attempts”. 4 https://www.lesswrong.com/feed.xml fnrh0odfi7xg Using interpretability tools, the authors conclude “RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones”. I think the basic idea is that, to do well in long-chain-of-thought reasoning, there are a bunch of moves that you can make: press on, backtrack, reexamine assumptions, pivot to Alternative Strategy X, etc. The LLM “knows” all those moves from imitative learning, but doesn’t have refined heuristics on which move to make in which situation. These kinds of heuristics are very important for making progress in a long CoT, and they’re somewhat domain-specific, and you can’t really pick them up well by imitative learning. So that’s the main low-hanging fruit that RLVR fixes, and it makes a huge difference in practice. Caveat: Just as in the previous section, this paper is old by RLVR standards , and not based on bleeding-edge frontier LLMs with bleeding-edge RLVR best practices. I guess in this case I’m slightly reassured by the fact that one of the coauthors, @Neel Nanda https://www.lesswrong.com/users/neel-nanda-1?mention=user , works with near-SOTA LLMs at his day job at DeepMind. But only slightly. Anyway, I’m taking this claim as a good default mental picture, not an ironclad eternal truth. In particular, I remain open-minded to the Liu et al. 2025 claim https://arxiv.org/abs/2505.24864 that “RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks.” But I’m also open-minded to the opposite Liu et al. make that claim but I don’t think they really prove it. For more discussion and references on this topic, see “Some Interesting Papers on RLVR” https://www.lesswrong.com/posts/tjwpdzpPcPbuwqmuP/some-interesting-papers-on-rlvr by @CarolusRenniusVitellius https://www.lesswrong.com/users/carolusrenniusvitellius?mention=user . What’s my point? I feel most strongly about the following intuitive picture : No question that the RL piece is big. It’s huge. It’s an elephant in the room. But if the RL piece is an elephant in the room, then the imitative learning piece is the friggin’ Mt. Everest in the room, if we’re trying to have a good mental picture of how LLMs can do the things that they can do. That’s all intuition. How do we move beyond that into a concrete operationalization? It might be interesting to look at KL divergence or weight changes between randomly-initialized models versus base models, and between base models versus RLVR? I can’t immediately find literature on that. Or we could talk about “bits of selection”. If something like the Yue et al. 2025 results above are valid, then imitative learning might get us from probability of accomplishing a certain task, to 0.1% probability. And then RL might get us from 0.1% to 90%. It would follow that almost all the “bits of selection” come from imitative learning 322 bits versus 10 bits, in this made-up example . See: “privileging the hypothesis” https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis . Yes I know, the Yue et al. 2025 results shouldn’t be taken at face value, but I think this is still the right bottom line. 6 Of course, to reiterate, all this is still compatible with RLVR being very important in a pragmatic sense: the difference between working 0.1% of the time versus 90% is a big deal Here’s three debates where this topic is relevant: There’s a nice explanation by @Rohin Shah https://www.lesswrong.com/users/rohinmshah?mention=user , on the 80,000 hours podcast recorded December 2025 https://80000hours.org/podcast/episodes/rohin-shah-google-deepmind-agi-safety/ how-much-longer-will-we-be-able-to-read-ais-thoughts-005417 : Rob Wiblin:…Just to check that I’ve understood, you’re saying because we are using GPUs and TPUs, structurally that forces the thought of the model to not be very deep: it can have many things in its mind at one time, but there can’t be very many steps, because you have to be going through all of these things in parallel. But if it was very deep , then you wouldn’t be able to do them all simultaneously; you would have to wait until the earlier steps were done to do the later steps. And this is just something that is going to remain the case for years to come. … Rohin Shah:Yeah, so that was stage number one, which is like, why should we expect the opaque serial depth to stay low, at least during the pretraining phase?Then there’s step two, which is like, we’re saying that the pre-trained model has to essentially speak in English in order to do reasoning. But then there’s a bunch of post-training and RL and all of this sort of stuff. Maybe that’s going to make it so that even if the model is speaking in tokens, maybe it will start speaking some sort of alien language that we don’t understand. And here I would say there’s no theoretical argument, or there’s no theorem you can prove that will say, no, they’re not going to do this. But I will say they are born speaking English in some sense, and pretraining is by far the most powerful form of getting stuff into an AI system that we have ever built. So the model is incredibly good at speaking in natural language. It’s great at doing reasoning, but only the kind of reasoning that humans do when writing stuff down. And then when you look at the reasoning training that we’re doing today, there’s a paper from Neel Nanda and others that shows that actually a substantial part of what the reasoning training is doing is just teaching the model when to do a specific kind of reasoning step that it had already learned during pretraining.So basically a lot of the capability is just coming in pretraining. We know that the pretraining, for good reason, is going to stay the way that it is, and be done using human-like reasoning and speaking in English. And then the RL is just really inefficient relative to pretraining, and for it to build an entirely new epistemic language that is something that we wouldn’t be able to understand even with a decent chunk of effort, is just so far beyond what RL is doing currently that it would be pretty surprising to see that in the near future. Now, do I expect to see it ever? Yes. But I think six months ago, I split a room in a conference by saying, “Agree or disagree? Chain of thought monitoring will continue for two years.” I think I said two years. It might have even been one year. I forget. Whereas my median is probably like four years, five years, I don’t know. It’s not totally clear, mostly because of this argument about how hard it would be to go beyond the current situation. This is kinda the mirror image of my §1.2 argument above. I was saying in §1.2: CoTs are legible, therefore LLM capabilities mostly come from imitative learning. Whereas Rohin is saying: LLM capabilities mostly come from imitative learning, therefore CoTs are legible, and this will not change anytime soon. Insofar as LLMs are powered by imitative learning, we should expect them to mainly be competent in domains where there is human data and human concepts to learn from. Insofar as LLMs are powered by RLVR, we should expect them to mainly be competent in domains where there are verifiable automatically-checkable rewards. I think both are relevant. And indeed, LLMs seem to be at their best when we have both. For example, software debugging allows both copying from abundant human concepts and human reasoning traces, and testing results in a controlled environment. By contrast, I have heard lots of people talk as if verifiable rewards are sufficient by themselves—that as long as we have verifiable rewards for such-and-such domain, LLMs will sooner or later be able to master that domain, even if the domain is arbitrarily far from anything humans have ever thought about. Well, that’s a hypothesis. Is it true? Seems to me that the jury is out; I don’t recall seeing demonstrations of such a thing to date. Anyway, I think my headline claim “LLMs are still mostly powered by imitative learning, not RL” provides some helpful background intuitions and context as we try to think through what to expect here. I think a very strong general rule-of-thumb is: if you create AI capabilities via RL, 7 https://www.lesswrong.com/feed.xml fnlcg3bvnw33h you get ruthless sociopathic behavior . See my post “Why we should expect ruthless sociopath ASI” https://www.lesswrong.com/posts/ZJZZEuPFKeEdkrRyf/why-we-should-expect-ruthless-sociopath-asi . The logic is basically: whatever the reward function is, reinforcement learning is reinforcing the ruthless and relentless pursuit of a high score on that function, by any available means. By contrast, if you create AI capabilities via imitative learning, you get models that follow the human distribution of outputs . Cf. “personas” https://alignment.anthropic.com/2026/psm/ , “simulators” https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators , etc. The human distribution has its problems, Lord knows, but it’s not nearly as cursed as the “100% ruthless sociopath all the time” that you normally get from building capabilities via pure RL training. For example, the Bing-Sidney https://www.lesswrong.com/w/microsoft-bing-sydney chatbot from 2023 was trained by pure imitative learning pretraining + SFT , without any RL at all. 8 Was it misaligned? Anyway, that’s my proposal, and I claim that it’s fully compatible with the alignment-related evidence that we’ve seen from LLMs to date. They are definitely not 100% ruthless sociopaths all the time. But they are somewhat like that sometimes . Correspondingly, RLVR is part of, but not most of, how LLMs get their capabilities today. If we want to refine that idea, we get into my “Foom & Doom” §2.3.5 picture https://www.lesswrong.com/posts/bnnKGSCHJghAvqPjS/foom-and-doom-2-technical-alignment-is-hard 2 3 5 Putting everything together LLMs are generally not scheming right now but I expect future AI to be disanalogous where LLMs have some niceness from imitative learning, and then the more you do RLVR, the more that niceness gets diluted away . There seem to be ways to mitigate the damage of RLVR e.g. inoculation prompting https://www.lesswrong.com/posts/AXRHzCPMv6ywCxCFp/inoculation-prompting-instructing-models-to-misbehave-at , but that’s still directionally what I strongly expect to happen. How fast does RLVR dilute away the niceness? Unclear. But if LLMs are still mostly powered by imitative learning, not RLVR, and yet this comparatively small amount of RLVR is already enough to cause obvious RL-style alignment problems, then that’s a bad sign for our ever-more-RLVR LLM future. There are a couple edge-cases in this breakdown, and the way we should handle those cases depends on what we’re ultimately trying to figure out. My interests are listed in §3, and in those contexts I would say: 1 I agree with Arthur Conmy’s take https://x.com/ArthurConmy/status/2051648364442202262 that Direct Preference Optimization DPO usually has more in common with imitative learning than RL, even though it’s sometimes lumped in with RLHF; 2 If Company A uses RLVR, and the CoTs get posted on the internet, and then Company B does SFT on those CoTs, then this should count as RL even though it’s structurally imitative learning. These edge-cases won’t really come up anyway; the paradigm case that I want to talk about is a big AI company doing pretraining on human-created internet text, SFT on their own proprietary data, and RLVR on their own proprietary training environments. Cf. the Karpathy quote https://x.com/karpathy/status/1835561952258723930?lang=en : “You can tell the RL is done properly when the models cease to speak English in their chain of thought”. How much money? Not sure. The total amount that LLM companies are spending on “data” is huge e.g. “$10 to $15 billion per LLM company ” https://x.com/MTSlive/status/2060447529133867339 per year , but that claim seems to involve a broad definition of the word “data”, that combines both imitative learning data pretraining & SFT and RL training environments. I can’t easily figure out the breakdown in spending between imitative learning versus RL. However, the imitative learning part definitely seems to be more than zero. E.g. a recent blog post by an industry insider https://willdepue.substack.com/p/a-stargate-for-data mentions “high-quality midtraining tokens”. The paper says “regardless of the number of attempts”, but that raises the obvious question “well, how many attempts did you try?” The paper does not make that clear, unless I missed it. Their graphs top out at pass@256 i.e., probability of succeeding at least once, given 256 tries , but they must have done a bunch of pass@256 runs to get a pass@256 probability. Based on the plots, maybe attempts total? A town I picked randomly by pointing at a world map. Gemini 3.1 Pro without internet access suggests Sabbia https://www.sabbia.hr/en-EN . I’d go As mentioned above, Liu et al. 2025 https://arxiv.org/abs/2505.24864 found tasks where RLVR succeeded but the base model pass rate was below , and perhaps far below. However, those tasks seem to be ones that require long answers with lots of steps; so in those cases we would also need to lower the starting random policy success probability, to something way below . So the bottom line would stay the same: imitative learning is responsible for far more of the “bits of selection” than RL. Note that RLHF is not really a “creating AI capabilities”; rather, conventional wisdom is that RLHF makes models dumber, not smarter, as a side-effect of making them nicer. Again, this post is mostly about RLVR and similar. Source: see this comment https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned?commentId=AAC8jKeDp6xqsZK2K by @gwern https://www.lesswrong.com/users/gwern?mention=user .