Why do OpenAI's GPT-2 weights beat mine? Part two: IFT dropout

https://arxiv.org/pdf/2101.03961 -- p11 -- Regularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.

Could that be it?! Model conf has dropout in it, so...

Drop rate was zero in the GPT-2 weights JSON files I had! Nothing in the paper but Raschka and also https://huggingface.co/docs/transformers/v4.48.2/en/model_doc/gpt2#transformers.GPT2Config

When I did the extended/two-epoch runs, I wound up doing many more epochs for the IFT.

OK, firstly let's do a run where we match the models' pre-train dropout.

Epochs in keeping with previous ones except for OpenAI small which took 4 rather than 2, and oddly "Cloud FineWeb, 8x H100 80 GiB" which took an extra epoch.

The ordering has swapped around a bit, but nothing drastic. All models are scoring a bit higher than they were, apart from OpenAI medium and small. So maybe signal? Small is still way ahead though.

As you'd expect, epochs the same for the models base trained with dropout. Much higher

for non-dropout base models. Models base trained with dropout seem to have been largely unaffected, as you'd expect -- points go up or down but pretty much within a range of +/-1. But all models that were base trained without performed catastrophically worse, with the exception of

1xrtx3090-stacked-interventions , which actually went up a bit (though within the noise).

Interesting.

Training epochs stable vs first run in this post for models base trained with no dropout, as you'd expect. Models trained with dropout all get fewer epochs apart from Cloud FineWeb, 8x B200 160 GiB, which is stable

OpenAI weights unharmed.

Other with-dropout models mostly harmed apart from 1xrtx3090-baseline

, Local FineWeb train.

Need a better comparitive table.