cd /news/large-language-models/the-exact-same-perplexity-chart-from… · home › topics › large-language-models › article
[ARTICLE · art-174367] src=micjm.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

The exact same perplexity chart, from data mix alone

A follow-up blog post demonstrates that varying the percentage of code in a pre-training data mix alone can reproduce a chart shaped like the one OpenAI published showing pre-training progress, which the author argues was confounded by code share in the corpus. The author had earlier published a post showing that changing the code percentage in the pre-training mix and evaluating log perplexity loss on code almost by definition yields a reduction in log perplexity loss.

recommendation

Control for the percentage of code in your pre-training mix before crediting perplexity gains on code benchmarks to scale or method.

more in playbook: Evals and testing for LLM apps
read1 min views1 publishedNov 22, 2026

Earlier today, I published a blog post whose point was that OpenAI’s recent post showing pre-training progress was confounded by the percent of code in their pre-training corpus.

In the post, I demonstrated that if you vary the percent of code in your pre-training mix and you only evaluate log perplexity loss on a set of code, then you almost by definition get a reduction in log perplexity loss.

After publishing, I realized I could do one better and generate something in the shape of the exact same chart:

── more in #large-language-models 4 stories · sorted by recency
── more on @openai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/the-exact-same-perpl…] indexed:0 read:1min 2026-11-22 · —