# Why do OpenAI's GPT-2 weights beat mine? Part two: IFT dropout > Source: > Published: 2026-08-14 19:00:00+00:00 https://arxiv.org/pdf/2101.03961 -- p11 -- Regularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks. Could that be it?! Model conf has dropout in it, so... Drop rate was zero in the GPT-2 weights JSON files I had! Nothing in the paper but Raschka and also https://huggingface.co/docs/transformers/v4.48.2/en/model_doc/gpt2#transformers.GPT2Config When I did the extended/two-epoch runs, I wound up doing many more epochs for the IFT. OK, firstly let's do a run where we match the models' pre-train dropout. | Test loss | Base dropout | IFT epochs | IFT score | IFT rank | | |---|---|---|---|---|---| | OpenAI weights: medium | 3.231442 | Yes | 2 | 42.43 | 1 | | JAX, overtrained one long epoch | 3.324953 | No | 3 | 19.35 | 4 | | JAX, overtrained two normal epochs | 3.326482 | No | 4 | 18.68 | 7 | | JAX, with MHA bias, no dropout | 3.418784 | No | 4 | 18.96 | 6 | | JAX, no MHA bias, no dropout | 3.420089 | No | 5 | 21.45 | 3 | | JAX, no MHA bias, with dropout | 3.476802 | Yes | 7 | 17.17 | 10 | | OpenAI weights: small | 3.499677 | Yes | 4 | 23.66 | 2 | `1xrtx3090-stacked-interventions` | 3.538161 | No | 4 | 14.06 | 13 | `8xa100m40-stacked-interventions-1` | 3.577761 | No | 4 | 10.51 | 17 | | Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 6 | 19.25 | 5 | `1xrtx3090-baseline` | 3.683835 | Yes | 6 | 13.37 | 14 | `8xa100m40-baseline` | 3.691526 | Yes | 4 | 14.25 | 12 | | Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 5 | 14.31 | 11 | | Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 4 | 11.48 | 16 | | Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 11.50 | 15 | | Local FineWeb train | 3.943522 | Yes | 7 | 9.09 | 18 | | Local FineWeb-Edu extended train | 4.134991 | Yes | 7 | 17.70 | 8 | | Local FineWeb-Edu train | 4.166892 | Yes | 7 | 17.55 | 9 | Epochs in keeping with previous ones except for OpenAI small which took 4 rather than 2, and oddly "Cloud FineWeb, 8x H100 80 GiB" which took an extra epoch. The ordering has swapped around a bit, but nothing drastic. All models are scoring a bit higher than they were, apart from OpenAI medium and small. So maybe signal? Small is still way ahead though. | Test loss | Base dropout | IFT epochs | IFT score | IFT rank | | |---|---|---|---|---|---| | OpenAI weights: medium | 3.231442 | Yes | 2 | 42.22 | 1 | | JAX, overtrained one long epoch | 3.324953 | No | 19 | 7.26 | 17 | | JAX, overtrained two normal epochs | 3.326482 | No | 16 | 12.71 | 13 | | JAX, with MHA bias, no dropout | 3.418784 | No | 13 | 13.09 | 11 | | JAX, no MHA bias, no dropout | 3.420089 | No | 20 | 5.20 | 18 | | JAX, no MHA bias, with dropout | 3.476802 | Yes | 7 | 18.00 | 4 | | OpenAI weights: small | 3.499677 | Yes | 4 | 23.94 | 2 | `1xrtx3090-stacked-interventions` | 3.538161 | No | 13 | 15.28 | 7 | `8xa100m40-stacked-interventions-1` | 3.577761 | No | 19 | 8.32 | 16 | | Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 6 | 20.77 | 3 | `1xrtx3090-baseline` | 3.683835 | Yes | 6 | 14.16 | 10 | `8xa100m40-baseline` | 3.691526 | Yes | 4 | 15.17 | 8 | | Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 5 | 14.86 | 9 | | Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 4 | 12.35 | 14 | | Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 12.99 | 12 | | Local FineWeb train | 3.943522 | Yes | 7 | 10.31 | 15 | | Local FineWeb-Edu extended train | 4.134991 | Yes | 7 | 17.33 | 6 | | Local FineWeb-Edu train | 4.166892 | Yes | 7 | 17.94 | 5 | As you'd expect, epochs the same for the models base trained with dropout. *Much* higher for non-dropout base models. Models base trained with dropout seem to have been largely unaffected, as you'd expect -- points go up or down but pretty much within a range of +/-1. But all models that were base trained without performed catastrophically worse, with the exception of `1xrtx3090-stacked-interventions` , which actually went up a bit (though within the noise). Interesting. | Test loss | Base dropout | IFT epochs | IFT score | IFT rank | | |---|---|---|---|---|---| | OpenAI weights: medium | 3.231442 | Yes | 2 | 42.81 | 1 | | JAX, overtrained one long epoch | 3.324953 | No | 3 | 18.62 | 5 | | JAX, overtrained two normal epochs | 3.326482 | No | 4 | 18.94 | 4 | | JAX, with MHA bias, no dropout | 3.418784 | No | 4 | 17.60 | 6 | | JAX, no MHA bias, no dropout | 3.420089 | No | 5 | 20.60 | 3 | | JAX, no MHA bias, with dropout | 3.476802 | Yes | 5 | 12.55 | 15 | | OpenAI weights: small | 3.499677 | Yes | 2 | 25.86 | 2 | `1xrtx3090-stacked-interventions` | 3.538161 | No | 4 | 13.15 | 14 | `8xa100m40-stacked-interventions-1` | 3.577761 | No | 4 | 10.11 | 18 | | Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 3 | 16.78 | 7 | `1xrtx3090-baseline` | 3.683835 | Yes | 4 | 14.94 | 8 | `8xa100m40-baseline` | 3.691526 | Yes | 3 | 13.71 | 12 | | Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 4 | 13.86 | 11 | | Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 3 | 10.75 | 17 | | Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 13.60 | 13 | | Local FineWeb train | 3.943522 | Yes | 5 | 12.03 | 16 | | Local FineWeb-Edu extended train | 4.134991 | Yes | 5 | 14.19 | 10 | | Local FineWeb-Edu train | 4.166892 | Yes | 5 | 14.75 | 9 | Training epochs stable vs first run in this post for models base trained with no dropout, as you'd expect. Models trained with dropout all get fewer epochs apart from Cloud FineWeb, 8x B200 160 GiB, which is stable OpenAI weights unharmed. Other with-dropout models mostly harmed apart from `1xrtx3090-baseline` , Local FineWeb train. Need a better comparitive table.