A developer extended Sebastian Raschka's GPT-2-style code from the book "Build a Large Language Model (from Scratch)" to add mixture-of-experts support and trained a 446M-parameter model with 220M act…
Researchers studying Looped GPT-BERT in the BabyLM 2026 Strict-small setting trained a 12.18M-parameter model on a preprocessed 7.48M-word English corpus, using four physical layers for twelve recurre…
A new arXiv paper (2609.05448v1) formulates structured post-training pruning of vision and language transformers as a damage-aware multi-armed bandit problem under a fixed candidate-evaluation budget,…
The Tokenizer newsletter #36 reports that a single C file can stream a 744-billion-parameter model from disk, Cloudflare has mapped agentic traffic patterns, and Andrej Karpathy's nanochat trains a GP…
A study by Epoch AI finds that from 2019 to 2025, data improvements contributed 12.0x compute efficiency gains compared to 3.7x from model improvements, meaning 3.24x more gains came from data at a 1e…
An agent harness is the code and structure around a language model that converts raw text prediction into goal-directed behavior, and it has driven larger performance gains than model upgrades, accord…
A new arXiv preprint (2609.03090v1) reports that matrix-CODI, a continuous chain-of-thought model with matrix-valued latent bottlenecks, shows rank-indifference on the ProsQA benchmark: rank-k project…
Developer gpjt has uploaded PyTorch-compatible versions of all his JAX-trained GPT-2 models to the Hugging Face Hub, including models from his blog series on writing an LLM from scratch and Chinchilla…
PicoLM v1.0-rc1, an LLM inference engine written in C99, has been released, supporting llama-2, GPT-2, Qwen 3.6/3.8(+MoE), and Gemma-3n models. The engine features CPU SIMD acceleration, CUDA/HIP supp…
Ben Houston's Three-LLM library runs LLMs entirely in the browser via Three.js and WebGPU, supporting GPT-2, Llama-style, Gemma 3, Phi, and Qwen3.5 architectures, with checkpoints from 3M to 1.3B para…
LLM tokenization, while a small part of overall latency, sits on the hot path and can occur multiple times per request, according to a technical analysis of GPT-2's reference encoder. The analysis sho…
A developer's analysis of pre-tokenization regexes shows that GPT-2's word-splitting pattern omitted Unicode's Mark category, a bug inherited by GPT-4, Llama 3, Qwen 3, and GLM-4/5, forcing BPE to tok…
In a technical blog post, an independent researcher investigating why OpenAI's GPT-2 small weights outperform their own models on an instruction-fine-tuning test discovered that their fine-tuning conf…
Latitude opened its AI roleplaying platform Voyage to the public on Wednesday, removing the waitlist for the open beta available on iOS, Android, and the web. The platform separates world state manage…
A developer found a scale-selection rule for NVFP4 gradient quantization that reduces mean squared error by 14% on real gradient tensors compared to the state-of-the-art MS-EDEN estimator, but trainin…
Even infants are far more efficient language learners than today's AI chatbots, according to experts interviewed by MIT Technology Review. Stanford cognitive scientist Michael C. Frank noted that todd…
Researchers introduced RODE, a matrix-aware optimizer that decouples radial and directional updates, outperforming Muon variants across language modeling and image classification tasks. At 1.5B scale,…
AlpinDale released gpt2.cmake, an implementation of OpenAI's GPT-2 language model written entirely in CMake, using Q16.16 fixed-point integer arithmetic to run inference without floating-point operati…
A new survey of positional encoding methods in Transformers argues that the field is best understood not as a chronological progression but as answers to a single question: where position information …
In a blog post, an unnamed author argues that AI performance will not plateau, countering the common belief that progress is driven by brute-force scale with diminishing returns. The author contends t…