Independent LLM-from-scratch experiments found that OpenAI's GPT-2 small (124M parameters) consistently outperformed the author's 163M-parameter models on an instruction fine-tuning task adapted from …
A developer extended Sebastian Raschka's GPT-2-style code from the book "Build a Large Language Model (from Scratch)" to add mixture-of-experts support and trained a 446M-parameter model with 220M act…
Developer gpjt has uploaded PyTorch-compatible versions of all his JAX-trained GPT-2 models to the Hugging Face Hub, including models from his blog series on writing an LLM from scratch and Chinchilla…
In a technical blog post, an independent researcher investigating why OpenAI's GPT-2 small weights outperform their own models on an instruction-fine-tuning test discovered that their fine-tuning conf…
Simon Willison added D2 diagram support to his static site generator, allowing him to compile D2 files into SVG diagrams for blog posts. The feature, implemented via a Python function that invokes the…
PyTorch's built-in GELU function is 20% faster than a hand-rolled version when training GPT-2 small models, according to a developer's benchmark. The same code training the same model on the same data…
OpenAI's GPT-2 medium weights achieved the highest IFT score of 42.43 with only 2 IFT epochs, while a JAX model with no dropout and no MHA bias scored 21.45 with 5 epochs, and a JAX model with dropout…
Giles Thomas, a developer, tested the Chinchilla scaling rule by comparing overtrained GPT-2 style models (trained on 40 tokens per parameter) against a model scaled up in parameters and tokens equall…
In a blog post, the author describes their personal policy for using AI tools like ChatGPT and Claude in creating content for their blog, emphasizing that AI is used for ideation, code review, and edi…
Giles Thomas, in a blog post, reports that deliberately overtraining GPT-2-style models did not improve their performance on instruction fine-tuning evaluations, contrary to his hypothesis that OpenAI…
A bug in the evaluation code for GPT-2 style models caused incorrect baseline numbers, but OpenAI's original weights still outperform the author's models on instruction-following tasks. The bug involv…
OpenAI's original GPT-2 small and medium weights consistently outperform custom-trained models in instruction-following evaluations, despite some custom models achieving better test loss, according to…
Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 using Unsloth's UD-IQ4_NL_XL quantisation achieved up to 140 tokens per second for generation and over 3,300 tok/s for prompt processing with a…
A developer building a GPT-2 implementation in JAX discovered that token embeddings and the output head account for nearly half of the model's 163 million parameters, while attention layers use fewer …
A developer repurposed an old small-form-factor PC named 'poppy' into a dedicated machine for local LLM training, upgrading its case and power supply to accommodate future multi-GPU setups. The projec…
Giles Thomas completed building and training a GPT-2 small model from scratch using JAX, achieving a test loss of 3.418784, outperforming both his PyTorch model (3.538161) and the original GPT-2 small…
Giles Thomas built a JAX training loop for an LLM from scratch, using NNX and Optax, as part of a project to replicate GPT-2 small quality without referencing Sebastian Raschka's book. He trained a ba…
Researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell found that large language models often ignore explicit role tags like <system> or <user> and instead infer roles from text tone, enabling…
A developer debugging a JAX/Flax NNX training loop discovered that the loss was stuck at 10.82, indicating the model was performing no better than random guessing. The issue was traced to the training…
JAX's default_device context manager places arrays on the specified device but does not commit them, allowing JAX to move them to other devices. This caused array lookups to take over a second by trig…