I, too, can make a chart of perplexity drops A blog post published by OpenAI claiming that perplexity drops can serve as a universal measure of intelligence drew a rebuttal pointing to a 2025 paper (arXiv 2403.16952) that trained identical 160M-parameter models on five mixes of GitHub code and web text and found perplexity on held-out code falls purely as the code share of the training corpus rises. The critic noted OpenAI's chart measures perplexity against a fixed corpus of code, and that OpenAI itself concedes in a footnote that "how well they predict code is a poor proxy for their general capabilities." The argument: perplexity loss is only meaningful relative to the text chosen to generate it, so OpenAI's post needs qualifications. I, too, can make a chart of perplexity drops Today, OpenAI released a blog post https://openai.com/index/perplexity-and-intelligence/ claiming that perplexity drops can be a universal measurement of intelligence. The core of the post is a chart showing perplexity values of their models dropping over time. The immediate question that comes to mind seeing the chart is: perplexity against what? The answer to this is buried in a footer. It’s perplexity against a fixed corpus of code. But the problem with this is that maybe the proportion of their pre-training mix that is code is increasing. Lowering perplexity without scaling So I set out to show that I could lower perplexity by just changing the pretraining mix. As I spun an agent up to get started with this, it helpfully pointed out that it had already been done in this 2025 paper https://arxiv.org/abs/2403.16952 . Funny enough, this was Astra 6.2, it certainly wasn’t perplexed by the task. The authors trained models on five mixes of GitHub code and web text, with the same architecture and the same model size 160M parameters , then tested each on held-out code: Spoiler: the “internal model” here is the same 160M model with 75% of training corpus being code. Raising perplexity without scaling As expected, the same effect holds in reverse. The same paper shows increases in performance on text only as the code share decreases. Conclusions The obvious rebuttal to all of this is that the 160M model with 75% code in training mix is almost certainly more intelligent in terms of code than the 25% model. So what’s my point? Only that perplexity loss only measures something meaningful in the context of what text is chosen to generate the loss. And that OpenAI’s post needs a few qualifications 1 . 1 Funny enough, OpenAI admits as such in a footnote, saying in reference to GPT-2 and GPT-3: “how well they predict code is a poor proxy for their general capabilities.” Why they didn’t redo this analysis with a more general corpus is anyone’s guess.