cd /news/artificial-intelligence/i-too-can-make-a-chart-of-perplexity… · home › topics › artificial-intelligence › article
[ARTICLE · art-174317] src=micjm.com ↗ pub= topic=artificial-intelligence verified=true sentiment=↓ negative

I, too, can make a chart of perplexity drops

A blog post published by OpenAI claiming that perplexity drops can serve as a universal measure of intelligence drew a rebuttal pointing to a 2025 paper (arXiv 2403.16952) that trained identical 160M-parameter models on five mixes of GitHub code and web text and found perplexity on held-out code falls purely as the code share of the training corpus rises. The critic noted OpenAI's chart measures perplexity against a fixed corpus of code, and that OpenAI itself concedes in a footnote that "how well they predict code is a poor proxy for their general capabilities." The argument: perplexity loss is only meaningful relative to the text chosen to generate it, so OpenAI's post needs qualifications.

recommendation

Report perplexity alongside the exact corpus and its mix, and re-run the comparison on a general-text corpus before treating a perplexity drop as a capability gain.

more in playbook: Evals and testing for LLM apps
read2 min views2 publishedNov 22, 2026

Today, OpenAI released a blog post claiming that perplexity drops can be a universal measurement of intelligence. The core of the post is a chart showing perplexity values of their models dropping over time.

The immediate question that comes to mind seeing the chart is: perplexity against what? The answer to this is buried in a footer. It’s perplexity against a fixed corpus of code.

But the problem with this is that maybe the proportion of their pre-training mix that is code is increasing.

Lowering perplexity without scaling #

So I set out to show that I could lower perplexity by just changing the pretraining mix. As I spun an agent up to get started with this, it helpfully pointed out that it had already been done in this 2025 paper. (Funny enough, this was Astra 6.2, it certainly wasn’t perplexed by the task.)

The authors trained models on five mixes of GitHub code and web text, with the same architecture and the same model size (160M parameters), then tested each on held-out code:

Spoiler: the “internal model” here is the same 160M model with 75% of training corpus being code.

Raising perplexity without scaling #

As expected, the same effect holds in reverse. The same paper shows increases in performance on text only as the code share decreases.

Conclusions #

The obvious rebuttal to all of this is that the 160M model with 75% code in training mix is almost certainly more intelligent in terms of code than the 25% model. So what’s my point? Only that perplexity loss only measures something meaningful in the context of what text is chosen to generate the loss. And that OpenAI’s post needs a few qualifications [1].

[1] Funny enough, OpenAI admits as such in a footnote, saying in reference to GPT-2 and GPT-3: “how well they predict code is a poor proxy for their general capabilities.” Why they didn’t redo this analysis with a more general corpus is anyone’s guess.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @openai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-too-can-make-a-cha…] indexed:0 read:2min 2026-11-22 · —