PyTorch's built-in GELU function is 20% faster than a hand-rolled version when training GPT-2 small models, according to a developer's benchmark. The same code training the same model on the same data…
Unigram, a new Rust crate released on crates.io, encodes bytes as words that each cost exactly one LLM token, making an N-byte value cost exactly N tokens under Claude and other models. The bijective …
Developer Akshay Muchaklavya built Forge, a custom deep learning framework written from scratch in C++, and used it to implement GPT-2 that matches Hugging Face's transformers token-for-token under gr…
NARE LABS, built by a 15-year-old, released TUPOI, an attention-free language model that replaces dense attention with a Symplectic Hamiltonian Integrator, achieving strictly O(1) memory with a fixed …
A new LessWrong post by Evan R. Murphy proposes applying a red team vs. blue team framework to AI evaluations, arguing that current evaluation methodologies fail to account for models that can subvert…
OpenAI's GPT-2 medium weights achieved the highest IFT score of 42.43 with only 2 IFT epochs, while a JAX model with no dropout and no MHA bias scored 21.45 with 5 epochs, and a JAX model with dropout…
Students at a U.S. public university in spring/summer 2026 see AI chatbots as mature technology with little recent improvement, according to instructor observations. The instructor reports that studen…
Marco Giancotti argues that generative AI's success stems from anthropomorphism, as large language models like OpenAI's GPT-2, which were originally next-word predictors, now create an irresistible il…
Researchers trained five GPT-2 architecture models from scratch to compare four monolingual Dravidian language models (Tamil, Telugu, Kannada, Malayalam) against one joint multilingual model sharing a…
A developer's optimization campaign on an RTX 3080 Ti cut a GPT-2-small-class transformer's forward pass from 78.2ms to 1.60ms, a 49× speedup, beating torch.compile's 1.72ms and reaching 136,000 token…
Software engineer and blogger Naoki Shibuya warns that developers who blindly trust AI-generated code risk becoming 'meat proxies' who add no value, and argues that critical thinking is the only diffe…
Giles Thomas, a developer, tested the Chinchilla scaling rule by comparing overtrained GPT-2 style models (trained on 40 tokens per parameter) against a model scaled up in parameters and tokens equall…
A developer building an LLM inference engine from scratch detailed the tokenization pipeline, which converts raw text into token IDs through normalization, regex pre-tokenization, byte-to-Unicode mapp…
Ninjahawk released bongochat, a 125.8-million-parameter language model trained end to end with nanochat on a single RTX 5070 (12GB) GPU in about 2.5 hours with no cloud spend, and it ranks #1 on the G…
In a pqc-forum discussion, Howard Y. Jung of Norma Co. claimed to have recovered a private key from an RSA-encrypted public key using a quantum simulator, utilizing 58 qubits and about 1 hour 54 minut…
A Baseten inference engineer known as @waterloo_intern published a technical blog post titled '22,580: From GPT-2 to Kimi K3, Explained,' which has garnered 2.4 million views. The post provides runnab…
Giles Thomas, in a blog post, reports that deliberately overtraining GPT-2-style models did not improve their performance on instruction fine-tuning evaluations, contrary to his hypothesis that OpenAI…
A developer fixed a GPT-2 reproducibility bug caused by missing std=0.02 in nn.init.normal_ for embedding and linear projection layers, which inflated initial weight variance and caused validation per…
A bug in the evaluation code for GPT-2 style models caused incorrect baseline numbers, but OpenAI's original weights still outperform the author's models on instruction-following tasks. The bug involv…
Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…