Finally, there is some information from one of the closed labs that backs up the fact that pre-training is not plateauing. (This is great and rare transparency from OpenAI.)
As I kept saying last year, pre-training is not plateauing, but it’s boring. Reinforcement learning with verifiable rewards (RLVR) for reasoning and agent harness was where most of the attention was.
Still, pre-training is super important to establish the foundation for later progress via RLVR and harness engineering. I.e.,
- pre-training equips the model with knowledge,
- and post-training (supervised fine-tuning and RLVR) lets the model access the knowledge
- and the harness lets the model apply that knowledge.
Note that the chart above reports log-perplexity, which is a next-token prediction metric (not a task-solving success metric). It’s fascinating how well it (still) correlates with perceived real-world performance, though.
(How do we compute perplexity? Take the sentence “I really like chocolate”. We give the model “I really like” and look up the probability it assigned to “chocolate” as the next token. We do that for each target token, take the negative natural log of each probability, and average the results. That’s the cross-entropy loss, or log-perplexity on the chart. (I have a longer section, with hands-on coding examples in both my LLMs from Scratch and Reasoning From Scratch books in case you are new to it and want to build a solid intuition and understanding.)
Now, I’d still love to see the same plot with training FLOPs on the x-axis.
Anyways, I expect the big labs to keep investing in pre-training. It won’t be the flashy and sexy work people will talk about (like RLVR in 2025 & 2026), but some of the gains may still come from this fairly boring work, like figuring out how much code to put in the training mix.