The exact same perplexity chart, from data mix alone A follow-up blog post demonstrates that varying the percentage of code in a pre-training data mix alone can reproduce a chart shaped like the one OpenAI published showing pre-training progress, which the author argues was confounded by code share in the corpus. The author had earlier published a post showing that changing the code percentage in the pre-training mix and evaluating log perplexity loss on code almost by definition yields a reduction in log perplexity loss. The exact same perplexity chart, from data mix alone Earlier today, I published a blog post https://micjm.com/blog/lowering-perplexity-without-scaling/ whose point was that OpenAI’s recent post showing pre-training progress was confounded by the percent of code in their pre-training corpus. In the post, I demonstrated that if you vary the percent of code in your pre-training mix and you only evaluate log perplexity loss on a set of code, then you almost by definition get a reduction in log perplexity loss. After publishing, I realized I could do one better and generate something in the shape of the exact same chart: