{"slug": "the-exact-same-perplexity-chart-from-data-mix-alone", "title": "The exact same perplexity chart, from data mix alone", "summary": "A follow-up blog post demonstrates that varying the percentage of code in a pre-training data mix alone can reproduce a chart shaped like the one OpenAI published showing pre-training progress, which the author argues was confounded by code share in the corpus. The author had earlier published a post showing that changing the code percentage in the pre-training mix and evaluating log perplexity loss on code almost by definition yields a reduction in log perplexity loss.", "body_md": "# The exact same perplexity chart, from data mix alone\n\nEarlier today, I published [a blog post](https://micjm.com/blog/lowering-perplexity-without-scaling/) whose point was that OpenAI’s recent post showing pre-training progress was confounded by the percent of code in their pre-training corpus.\n\nIn the post, I demonstrated that if you vary the percent of code in your pre-training mix and you only evaluate log perplexity loss on a set of code, then you almost by definition get a reduction in log perplexity loss.\n\nAfter publishing, I realized I could do one better and generate something in the shape of the exact same chart:", "url": "https://wpnews.pro/news/the-exact-same-perplexity-chart-from-data-mix-alone", "canonical_source": "https://micjm.com/blog/same-perplexity-chart-data-mix/", "published_at": "2026-11-22 23:05:36+00:00", "updated_at": "2026-11-22 23:11:45.312113+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research"], "entities": ["OpenAI"], "also_reported_by": [], "takeaway": "Varying only the percentage of code in a pre-training data mix can produce a chart shaped like OpenAI's pre-training progress chart, so code-share confounds log perplexity loss measured on code.", "recommendation": {"advice": "Control for the percentage of code in your pre-training mix before crediting perplexity gains on code benchmarks to scale or method.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/the-exact-same-perplexity-chart-from-data-mix-alone", "markdown": "https://wpnews.pro/news/the-exact-same-perplexity-chart-from-data-mix-alone.md", "text": "https://wpnews.pro/news/the-exact-same-perplexity-chart-from-data-mix-alone.txt", "jsonld": "https://wpnews.pro/news/the-exact-same-perplexity-chart-from-data-mix-alone.jsonld"}}