Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtraining
Giles Thomas, in a blog post, reports that deliberately overtraining GPT-2-style models did not improve their performance on instruction fine-tuning evaluations, contrary to his hypothesis that OpenAI's models were overt…