Why do OpenAI's GPT-2 weights beat mine?  Part two: IFT dropout

OpenAI's GPT-2 medium weights achieved the highest IFT score of 42.43 with only 2 IFT epochs, while a JAX model with no dropout and no MHA bias scored 21.45 with 5 epochs, and a JAX model with dropout scored 17.17 with 7 epochs, according to a comparison table from the author's experiments. The author found that their GPT-2 weight JSON files had zero dropout, contrary to the paper's mention of dropout, and tested matching pre-train dropout, but the ranking remained largely unchanged except for slight score increases.

https://arxiv.org/pdf/2101.03961 -- p11 -- Regularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. 2019 use dropout Srivastava et al., 2014 at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks. Could that be it? Model conf has dropout in it, so... Drop rate was zero in the GPT-2 weights JSON files I had Nothing in the paper but Raschka and also https://huggingface.co/docs/transformers/v4.48.2/en/model doc/gpt2 transformers.GPT2Config When I did the extended/two-epoch runs, I wound up doing many more epochs for the IFT. OK, firstly let's do a run where we match the models' pre-train dropout. | Test loss | Base dropout | IFT epochs | IFT score | IFT rank | | |---|---|---|---|---|---| | OpenAI weights: medium | 3.231442 | Yes | 2 | 42.43 | 1 | | JAX, overtrained one long epoch | 3.324953 | No | 3 | 19.35 | 4 | | JAX, overtrained two normal epochs | 3.326482 | No | 4 | 18.68 | 7 | | JAX, with MHA bias, no dropout | 3.418784 | No | 4 | 18.96 | 6 | | JAX, no MHA bias, no dropout | 3.420089 | No | 5 | 21.45 | 3 | | JAX, no MHA bias, with dropout | 3.476802 | Yes | 7 | 17.17 | 10 | | OpenAI weights: small | 3.499677 | Yes | 4 | 23.66 | 2 | 1xrtx3090-stacked-interventions | 3.538161 | No | 4 | 14.06 | 13 | 8xa100m40-stacked-interventions-1 | 3.577761 | No | 4 | 10.51 | 17 | | Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 6 | 19.25 | 5 | 1xrtx3090-baseline | 3.683835 | Yes | 6 | 13.37 | 14 | 8xa100m40-baseline | 3.691526 | Yes | 4 | 14.25 | 12 | | Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 5 | 14.31 | 11 | | Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 4 | 11.48 | 16 | | Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 11.50 | 15 | | Local FineWeb train | 3.943522 | Yes | 7 | 9.09 | 18 | | Local FineWeb-Edu extended train | 4.134991 | Yes | 7 | 17.70 | 8 | | Local FineWeb-Edu train | 4.166892 | Yes | 7 | 17.55 | 9 | Epochs in keeping with previous ones except for OpenAI small which took 4 rather than 2, and oddly "Cloud FineWeb, 8x H100 80 GiB" which took an extra epoch. The ordering has swapped around a bit, but nothing drastic. All models are scoring a bit higher than they were, apart from OpenAI medium and small. So maybe signal? Small is still way ahead though. | Test loss | Base dropout | IFT epochs | IFT score | IFT rank | | |---|---|---|---|---|---| | OpenAI weights: medium | 3.231442 | Yes | 2 | 42.22 | 1 | | JAX, overtrained one long epoch | 3.324953 | No | 19 | 7.26 | 17 | | JAX, overtrained two normal epochs | 3.326482 | No | 16 | 12.71 | 13 | | JAX, with MHA bias, no dropout | 3.418784 | No | 13 | 13.09 | 11 | | JAX, no MHA bias, no dropout | 3.420089 | No | 20 | 5.20 | 18 | | JAX, no MHA bias, with dropout | 3.476802 | Yes | 7 | 18.00 | 4 | | OpenAI weights: small | 3.499677 | Yes | 4 | 23.94 | 2 | 1xrtx3090-stacked-interventions | 3.538161 | No | 13 | 15.28 | 7 | 8xa100m40-stacked-interventions-1 | 3.577761 | No | 19 | 8.32 | 16 | | Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 6 | 20.77 | 3 | 1xrtx3090-baseline | 3.683835 | Yes | 6 | 14.16 | 10 | 8xa100m40-baseline | 3.691526 | Yes | 4 | 15.17 | 8 | | Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 5 | 14.86 | 9 | | Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 4 | 12.35 | 14 | | Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 12.99 | 12 | | Local FineWeb train | 3.943522 | Yes | 7 | 10.31 | 15 | | Local FineWeb-Edu extended train | 4.134991 | Yes | 7 | 17.33 | 6 | | Local FineWeb-Edu train | 4.166892 | Yes | 7 | 17.94 | 5 | As you'd expect, epochs the same for the models base trained with dropout. Much higher for non-dropout base models. Models base trained with dropout seem to have been largely unaffected, as you'd expect -- points go up or down but pretty much within a range of +/-1. But all models that were base trained without performed catastrophically worse, with the exception of 1xrtx3090-stacked-interventions , which actually went up a bit though within the noise . Interesting. | Test loss | Base dropout | IFT epochs | IFT score | IFT rank | | |---|---|---|---|---|---| | OpenAI weights: medium | 3.231442 | Yes | 2 | 42.81 | 1 | | JAX, overtrained one long epoch | 3.324953 | No | 3 | 18.62 | 5 | | JAX, overtrained two normal epochs | 3.326482 | No | 4 | 18.94 | 4 | | JAX, with MHA bias, no dropout | 3.418784 | No | 4 | 17.60 | 6 | | JAX, no MHA bias, no dropout | 3.420089 | No | 5 | 20.60 | 3 | | JAX, no MHA bias, with dropout | 3.476802 | Yes | 5 | 12.55 | 15 | | OpenAI weights: small | 3.499677 | Yes | 2 | 25.86 | 2 | 1xrtx3090-stacked-interventions | 3.538161 | No | 4 | 13.15 | 14 | 8xa100m40-stacked-interventions-1 | 3.577761 | No | 4 | 10.11 | 18 | | Cloud FineWeb, 8x A100 40 GiB | 3.673623 | Yes | 3 | 16.78 | 7 | 1xrtx3090-baseline | 3.683835 | Yes | 4 | 14.94 | 8 | 8xa100m40-baseline | 3.691526 | Yes | 3 | 13.71 | 12 | | Cloud FineWeb, 8x H100 80 GiB | 3.724507 | Yes | 4 | 13.86 | 11 | | Cloud FineWeb, 8x A100 80 GiB | 3.729900 | Yes | 3 | 10.75 | 17 | | Cloud FineWeb, 8x B200 160 GiB | 3.771478 | Yes | 4 | 13.60 | 13 | | Local FineWeb train | 3.943522 | Yes | 5 | 12.03 | 16 | | Local FineWeb-Edu extended train | 4.134991 | Yes | 5 | 14.19 | 10 | | Local FineWeb-Edu train | 4.166892 | Yes | 5 | 14.75 | 9 | Training epochs stable vs first run in this post for models base trained with no dropout, as you'd expect. Models trained with dropout all get fewer epochs apart from Cloud FineWeb, 8x B200 160 GiB, which is stable OpenAI weights unharmed. Other with-dropout models mostly harmed apart from 1xrtx3090-baseline , Local FineWeb train. Need a better comparitive table.