The NanoGPT speedrun repository reports that a collaborative effort has trained a language model to 3.28 cross-entropy loss on the FineWeb validation set in under 75 seconds on 8 NVIDIA H100 GPUs, a d…
A new model compression method called requential coding compresses a generative model by coding a training process built from self-generated data, achieving code length equal to the cumulative teacher…
Researchers trained decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, producing monthly point-in-time language model checkpoints fro…
A researcher reproduced Anthropic's paper on natural language activations (NLAs), training models to generate textual descriptions of neural network activations. The study found that a single model tr…
A developer built a novel triple-hybrid LLM combining Mamba, Attention, and a 32-expert Mixture of Experts architecture from scratch for approximately $50, completing Titan v1 and the first training c…
A tutorial demonstrates streaming, filtering, deduplication, tokenization, and analytics on the FineWeb dataset using Python, reproducing quality-filtering pipelines and MinHash-based near-duplicate d…
A team of researchers from Yale's Building AI Infrastructure class achieved a new speed record of 81.2 seconds on the modded-nanoGPT benchmark by introducing a learnable per-(layer, head) tanh(α) gate…