{"slug": "why-do-openai-s-gpt-2-weights-beat-mine-part-two-ift-dropout", "title": "Why do OpenAI's GPT-2 weights beat mine?  Part two: IFT dropout", "summary": "OpenAI's GPT-2 medium weights achieved the highest IFT score of 42.43 with only 2 IFT epochs, while a JAX model with no dropout and no MHA bias scored 21.45 with 5 epochs, and a JAX model with dropout scored 17.17 with 7 epochs, according to a comparison table from the author's experiments. The author found that their GPT-2 weight JSON files had zero dropout, contrary to the paper's mention of dropout, and tested matching pre-train dropout, but the ranking remained largely unchanged except for slight score increases.", "body_md": "https://arxiv.org/pdf/2101.03961 -- p11 --\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nCould that be it?! Model conf has dropout in it, so...\n\nDrop rate was zero in the GPT-2 weights JSON files I had! Nothing in the paper but Raschka and also https://huggingface.co/docs/transformers/v4.48.2/en/model_doc/gpt2#transformers.GPT2Config\n\nWhen I did the extended/two-epoch runs, I wound up doing many more epochs for the IFT.\n\nOK, firstly let's do a run where we match the models' pre-train dropout.\n\n| Test loss | Base dropout | IFT epochs | IFT score | IFT rank | |\n|---|---|---|---|---|---|\n| OpenAI weights: medium | 3.231442 | Yes | 2 | 42.43 | 1 |\n| JAX, overtrained one long epoch | 3.324953 | No | 3 | 19.35 | 4 |\n| JAX, overtrained two normal epochs | 3.326482 | No | 4 | 18.68 | 7 |\n| JAX, with MHA bias, no dropout | 3.418784 | No | 4 | 18.96 | 6 |\n| JAX, no MHA bias, no dropout | 3.420089 | No | 5 | 21.45 | 3 |\n| JAX, no MHA bias, with dropout | 3.476802 | Yes | 7 | 17.17 | 10 |\n| OpenAI weights: small | 3.499677 | Yes | 4 | 23.66 | 2 |\n`1xrtx3090-stacked-interventions` |\n3.538161 | No | 4 | 14.06 | 13 |\n`8xa100m40-stacked-interventions-1` |\n3.577761 | No | 4 | 10.51 | 17 |\n| Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 6 | 19.25 | 5 |\n`1xrtx3090-baseline` |\n3.683835 | Yes | 6 | 13.37 | 14 |\n`8xa100m40-baseline` |\n3.691526 | Yes | 4 | 14.25 | 12 |\n| Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 5 | 14.31 | 11 |\n| Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 4 | 11.48 | 16 |\n| Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 11.50 | 15 |\n| Local FineWeb train | 3.943522 | Yes | 7 | 9.09 | 18 |\n| Local FineWeb-Edu extended train | 4.134991 | Yes | 7 | 17.70 | 8 |\n| Local FineWeb-Edu train | 4.166892 | Yes | 7 | 17.55 | 9 |\n\nEpochs in keeping with previous ones except for OpenAI small which took 4 rather than 2, and oddly \"Cloud FineWeb, 8x H100 80 GiB\" which took an extra epoch.\n\nThe ordering has swapped around a bit, but nothing drastic. All models are scoring a bit higher than they were, apart from OpenAI medium and small. So maybe signal? Small is still way ahead though.\n\n| Test loss | Base dropout | IFT epochs | IFT score | IFT rank | |\n|---|---|---|---|---|---|\n| OpenAI weights: medium | 3.231442 | Yes | 2 | 42.22 | 1 |\n| JAX, overtrained one long epoch | 3.324953 | No | 19 | 7.26 | 17 |\n| JAX, overtrained two normal epochs | 3.326482 | No | 16 | 12.71 | 13 |\n| JAX, with MHA bias, no dropout | 3.418784 | No | 13 | 13.09 | 11 |\n| JAX, no MHA bias, no dropout | 3.420089 | No | 20 | 5.20 | 18 |\n| JAX, no MHA bias, with dropout | 3.476802 | Yes | 7 | 18.00 | 4 |\n| OpenAI weights: small | 3.499677 | Yes | 4 | 23.94 | 2 |\n`1xrtx3090-stacked-interventions` |\n3.538161 | No | 13 | 15.28 | 7 |\n`8xa100m40-stacked-interventions-1` |\n3.577761 | No | 19 | 8.32 | 16 |\n| Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 6 | 20.77 | 3 |\n`1xrtx3090-baseline` |\n3.683835 | Yes | 6 | 14.16 | 10 |\n`8xa100m40-baseline` |\n3.691526 | Yes | 4 | 15.17 | 8 |\n| Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 5 | 14.86 | 9 |\n| Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 4 | 12.35 | 14 |\n| Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 12.99 | 12 |\n| Local FineWeb train | 3.943522 | Yes | 7 | 10.31 | 15 |\n| Local FineWeb-Edu extended train | 4.134991 | Yes | 7 | 17.33 | 6 |\n| Local FineWeb-Edu train | 4.166892 | Yes | 7 | 17.94 | 5 |\n\nAs you'd expect, epochs the same for the models base trained with dropout. *Much* higher\nfor non-dropout base models.\n\nModels base trained with dropout seem to have been largely unaffected, as you'd expect --\npoints go up or down but pretty much within a range of +/-1. But all models that\nwere base trained without performed catastrophically worse, with the exception of\n`1xrtx3090-stacked-interventions`\n\n, which actually went up a bit (though within the noise).\n\nInteresting.\n\n| Test loss | Base dropout | IFT epochs | IFT score | IFT rank | |\n|---|---|---|---|---|---|\n| OpenAI weights: medium | 3.231442 | Yes | 2 | 42.81 | 1 |\n| JAX, overtrained one long epoch | 3.324953 | No | 3 | 18.62 | 5 |\n| JAX, overtrained two normal epochs | 3.326482 | No | 4 | 18.94 | 4 |\n| JAX, with MHA bias, no dropout | 3.418784 | No | 4 | 17.60 | 6 |\n| JAX, no MHA bias, no dropout | 3.420089 | No | 5 | 20.60 | 3 |\n| JAX, no MHA bias, with dropout | 3.476802 | Yes | 5 | 12.55 | 15 |\n| OpenAI weights: small | 3.499677 | Yes | 2 | 25.86 | 2 |\n`1xrtx3090-stacked-interventions` |\n3.538161 | No | 4 | 13.15 | 14 |\n`8xa100m40-stacked-interventions-1` |\n3.577761 | No | 4 | 10.11 | 18 |\n| Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 3 | 16.78 | 7 |\n`1xrtx3090-baseline` |\n3.683835 | Yes | 4 | 14.94 | 8 |\n`8xa100m40-baseline` |\n3.691526 | Yes | 3 | 13.71 | 12 |\n| Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 4 | 13.86 | 11 |\n| Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 3 | 10.75 | 17 |\n| Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 13.60 | 13 |\n| Local FineWeb train | 3.943522 | Yes | 5 | 12.03 | 16 |\n| Local FineWeb-Edu extended train | 4.134991 | Yes | 5 | 14.19 | 10 |\n| Local FineWeb-Edu train | 4.166892 | Yes | 5 | 14.75 | 9 |\n\nTraining epochs stable vs first run in this post for models base trained with no dropout, as you'd expect. Models trained with dropout all get fewer epochs apart from Cloud FineWeb, 8x B200 160 GiB, which is stable\n\nOpenAI weights unharmed.\n\nOther with-dropout models mostly harmed apart from `1xrtx3090-baseline`\n\n, Local FineWeb train.\n\nNeed a better comparitive table.", "url": "https://wpnews.pro/news/why-do-openai-s-gpt-2-weights-beat-mine-part-two-ift-dropout", "canonical_source": "https://www.gilesthomas.com/2026/08/why-do-openai-gpt2-weights-beat-mine-2-ift-dropout", "published_at": "2026-08-14 19:00:00+00:00", "updated_at": "2026-08-20 01:12:44.666596+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models"], "entities": ["OpenAI", "GPT-2", "JAX", "Hugging Face", "Raffel et al.", "Srivastava et al.", "Switch Transformers"], "alternates": {"html": "https://wpnews.pro/news/why-do-openai-s-gpt-2-weights-beat-mine-part-two-ift-dropout", "markdown": "https://wpnews.pro/news/why-do-openai-s-gpt-2-weights-beat-mine-part-two-ift-dropout.md", "text": "https://wpnews.pro/news/why-do-openai-s-gpt-2-weights-beat-mine-part-two-ift-dropout.txt", "jsonld": "https://wpnews.pro/news/why-do-openai-s-gpt-2-weights-beat-mine-part-two-ift-dropout.jsonld"}}