Giles Thomas, a developer, tested the Chinchilla scaling rule by comparing overtrained GPT-2 style models (trained on 40 tokens per parameter) against a model scaled up in parameters and tokens equall…
Giles Thomas, in a blog post, reports that deliberately overtraining GPT-2-style models did not improve their performance on instruction fine-tuning evaluations, contrary to his hypothesis that OpenAI…
OpenAI's original GPT-2 small and medium weights consistently outperform custom-trained models in instruction-following evaluations, despite some custom models achieving better test loss, according to…
Giles Thomas completed building and training a GPT-2 small model from scratch using JAX, achieving a test loss of 3.418784, outperforming both his PyTorch model (3.538161) and the original GPT-2 small…
Giles Thomas built a JAX training loop for an LLM from scratch, using NNX and Optax, as part of a project to replicate GPT-2 small quality without referencing Sebastian Raschka's book. He trained a ba…