Forethought researcher Tom Davidson's model of a software intelligence explosion, revised in a new Forethought article, finds it less likely that 10 or more years of total AI progress could fit into 1…
A new study of OLMo3 and Apertus pre-training checkpoints finds that language models repeatedly and abruptly switch between pattern-matching and generalizable computation, a behavior the authors call …
Training GPT-2 required roughly 10²¹ FLOPs, a level of compute that the 2004–2008 IBM BlueGene/L supercomputer at Lawrence Livermore National Laboratory could have delivered in about 41 days, accordin…
A study of sequence weighting across in-house and open-weight language models found non-monotonic scaling behavior: as models grow from small to medium scale they shift from learning general patterns …
Researchers introduced speculative sampling, an algorithm that accelerates transformer decoding by generating multiple tokens per call using a faster draft model and modified rejection sampling, achie…
Z.ai CEO Jie Tang announced that GLM 5.3's performance gains come from reinforcement learning on long-horizon environments, not parameter count, and introduced a new post-training scaling law with fiv…
Scaling machine learning models is more complex than simply increasing size, requiring careful engineering to avoid pitfalls, according to a technical essay by Beren Millidge. The author explains that…
A developer's analysis of scaling laws for large language models shows that the Chinchilla rule of about 20 training tokens per parameter is often treated as a fixed target, but recent scrutiny sugges…
Giles Thomas, a developer, tested the Chinchilla scaling rule by comparing overtrained GPT-2 style models (trained on 40 tokens per parameter) against a model scaled up in parameters and tokens equall…
Giles Thomas, in a blog post, reports that deliberately overtraining GPT-2-style models did not improve their performance on instruction fine-tuning evaluations, contrary to his hypothesis that OpenAI…
Google researchers have found that scaling laws similar to those for large language models apply to wearable sensor data, with validation loss scaling predictably with compute, model size, and data si…
AI teams often hit a 'latent capability ceiling' where larger models no longer improve task accuracy, with scaling data showing diminishing returns beyond 100B parameters and 61% of tasks exhibiting n…
A developer built a novel triple-hybrid LLM combining Mamba, Attention, and a 32-expert Mixture of Experts architecture from scratch for approximately $50, completing Titan v1 and the first training c…
Microsoft's MAI-Thinking-1 report introduces an "efficiency gain" (EG) metric that compares candidate model architectures against a baseline by measuring how much compute is needed to reach the same l…
Researchers have developed a conditional scaling law that predicts optimal large language model architectures for inference efficiency, training over 200 models from 80 million to 3 billion parameters…
A new analysis argues that the era of AI scaling producing clearly better models is ending, with frontier labs facing exponentially increasing costs for linear returns. The author claims that the inte…
Amazon researchers have developed a new scaling law that connects large language model architecture directly to inference efficiency, enabling faster text generation without sacrificing accuracy. The …