{"slug": "i-too-can-make-a-chart-of-perplexity-drops", "title": "I, too, can make a chart of perplexity drops", "summary": "A blog post published by OpenAI claiming that perplexity drops can serve as a universal measure of intelligence drew a rebuttal pointing to a 2025 paper (arXiv 2403.16952) that trained identical 160M-parameter models on five mixes of GitHub code and web text and found perplexity on held-out code falls purely as the code share of the training corpus rises. The critic noted OpenAI's chart measures perplexity against a fixed corpus of code, and that OpenAI itself concedes in a footnote that \"how well they predict code is a poor proxy for their general capabilities.\" The argument: perplexity loss is only meaningful relative to the text chosen to generate it, so OpenAI's post needs qualifications.", "body_md": "# I, too, can make a chart of perplexity drops\n\nToday, OpenAI released [a blog post](https://openai.com/index/perplexity-and-intelligence/) claiming that perplexity drops can be a universal measurement of intelligence. The core of the post is a chart showing perplexity values of their models dropping over time.\n\nThe immediate question that comes to mind seeing the chart is: perplexity against what? The answer to this is buried in a footer. It’s perplexity against a fixed corpus of code.\n\nBut the problem with this is that maybe the proportion of their pre-training mix that is code is increasing.\n\n## Lowering perplexity without scaling\n\nSo I set out to show that I could lower perplexity by just changing the pretraining mix. As I spun an agent up to get started with this, it helpfully pointed out that it had already been done in [this 2025 paper](https://arxiv.org/abs/2403.16952). (Funny enough, this was Astra 6.2, it certainly wasn’t perplexed by the task.)\n\nThe authors trained models on five mixes of GitHub code and web text, with the same architecture and the same model size (160M parameters), then tested each on held-out code:\n\nSpoiler: the “internal model” here is the same 160M model with 75% of training corpus being code.\n\n## Raising perplexity without scaling\n\nAs expected, the same effect holds in reverse. The same paper shows increases in performance on text only as the code share decreases.\n\n## Conclusions\n\nThe obvious rebuttal to all of this is that the 160M model with 75% code in training mix is almost certainly more intelligent in terms of code than the 25% model. So what’s my point? Only that perplexity loss only measures something meaningful in the context of what text is chosen to generate the loss. And that OpenAI’s post needs a few qualifications [1].\n\n[1] Funny enough, OpenAI admits as such in a footnote, saying in reference to GPT-2 and GPT-3: “how well they predict code is a poor proxy for their general capabilities.” Why they didn’t redo this analysis with a more general corpus is anyone’s guess.", "url": "https://wpnews.pro/news/i-too-can-make-a-chart-of-perplexity-drops", "canonical_source": "https://micjm.com/blog/lowering-perplexity-without-scaling/", "published_at": "2026-11-22 19:58:44+00:00", "updated_at": "2026-11-22 20:11:53.413133+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["OpenAI", "arXiv 2403.16952", "GPT-2", "GPT-3", "Astra 6.2"], "also_reported_by": [], "takeaway": "Perplexity is only comparable across models when measured on the same corpus mix: a 160M model trained on 75% code beats a 25%-code model on held-out code, so falling code perplexity can reflect data mix rather than general capability.", "recommendation": {"advice": "Report perplexity alongside the exact corpus and its mix, and re-run the comparison on a general-text corpus before treating a perplexity drop as a capability gain.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/i-too-can-make-a-chart-of-perplexity-drops", "markdown": "https://wpnews.pro/news/i-too-can-make-a-chart-of-perplexity-drops.md", "text": "https://wpnews.pro/news/i-too-can-make-a-chart-of-perplexity-drops.txt", "jsonld": "https://wpnews.pro/news/i-too-can-make-a-chart-of-perplexity-drops.jsonld"}}