In a 2020 analysis of Fred Brooks' 1986 essay 'No Silver Bullet,' the author argues that Brooks' claim of a 2x limit on programmer productivity improvements is flawed. The author contends that Brooks …
A 124M-parameter GPT-2 model trained from scratch on OpenWebText data using a custom deep learning library achieved a validation loss of 2.764 nats and a perplexity of 15.87 after 56,000 steps (27.5B …
Researchers at arXiv have identified a conflict signature in language models that distinguishes deceptive outputs from honest errors, achieving 100% accuracy in detecting lies across multiple models i…
Researchers tested whether language models can independently discover the concept of zero through arithmetic generalization. GPT-2-sized models failed at test time regardless of language pretraining, …
A developer debugging a JAX/Flax NNX training loop discovered that the loss was stuck at 10.82, indicating the model was performing no better than random guessing. The issue was traced to the training…
A new measurement framework reveals that eight open-source transformer LLMs partition into two distinct clusters based on runtime probability distribution dynamics, with an order-of-magnitude gap in G…
A tutorial demonstrates streaming, filtering, deduplication, tokenization, and analytics on the FineWeb dataset using Python, reproducing quality-filtering pipelines and MinHash-based near-duplicate d…
Researchers have introduced Rotary Value Embeddings (RoVE), a parameter-free modification to Rotary Position Embeddings (RoPE) that makes value pathways position-sensitive by rotating values simultane…
Google released Diffusion Gemma in early 2025 as its first open-weight diffusion language model, using a masked diffusion approach that generates text by starting with noise and iteratively refining i…
Mike X Cohen's new book "50 ML Projects to Understand LLMs" uses GPT-2 as a scientific specimen, teaching readers to investigate the model through 50 hands-on projects focused on code, statistics, and…
Researchers introduced trajectory extrapolation error, a measure of how much a language model's hidden states deviate from a linear path during word processing, and found it independently predicts hum…
Researchers have developed a bilayer SIR/SIRS epidemiological model to analyze how AI models collapse when trained on synthetic data generated by other AI systems, treating data corpora and AI models …
Researchers trained a small GPT-2 model on arithmetic problems using an Indonesian pedagogy called GASING, which breaks down calculations into left-to-right steps aligned with token generation. The 86…
DeepSpeed has integrated the Muon Optimizer, a memory-efficient optimizer that uses a single momentum buffer and Newton-Schulz orthogonalization to improve training convergence, particularly for 2D we…
Chinese AI labs have progressed from early BERT-era models to trillion-parameter systems like Wu Dao 2.0 (1.75T parameters) and cost-efficient architectures such as DeepSeek V3 (trained for $5.6M), ac…
Ablating induction heads in the GPT-2 small language model increased local repetition in the model's output, a finding validated through activation patching and comparison with random head ablation. T…
A user reported that Claude Opus 4.8 is exhibiting severe performance degradation, including an inability to read files, hallucinating file paths, and repeatedly generating errors with incorrect comma…
Epoch AI projects that compute power for running AI models will surpass compute power for building them by 2030, with nearly half of inference operations shifting to specialized ASICs while training c…
Shalin and Kanyes, cofounders of Hyper, launched a self-serve version of their "second brain" AI software that integrates personal context to automate tasks. The founders, who previously built robots …
NVIDIA released Nemotron-Labs Diffusion on May 23, 2026, a family of diffusion language models (DLMs) that generate entire blocks of tokens in parallel and iteratively refine them, rather than produci…