A new analysis compares large language models to sea squirts, arguing that post-training optimization for specific tool shapes is creating a Darwinian niche where models shape their own environment. T…
Researchers released the official Multi-Teacher On-Policy Distillation (MOPD) paper, which composes multiple capabilities into a single large language model by training domain-expert teachers independ…
Arena, an AI evaluation platform born at UC Berkeley, reached a $100M annual revenue run rate eight months after launching its product, as demand surges for benchmarks that measure real-world AI utili…
Scaling laws predict training loss as model size, dataset size, and compute scale, but their practical application is sensitive to hyperparameter choices like learning rate. Lilian Weng's post highlig…
Meta's LLMs have evolved from simple Transformer stacks to complex architectures with multiple attention variants, mixture-of-experts, and multimodal encoders, mirroring the complexity of recommendati…
A new benchmark called FactWorld reveals that hybrid AI models combining transformer and recurrent architectures can simultaneously excel at both associative recall and state tracking, capabilities th…
Microsoft AI released a detailed technical report on the development of its first model, MAI-Thinking-1, emphasizing a controlled, reproducible training process built on human-generated data and propr…
ServiceNow researchers have developed a new distillation method called π-Distill that allows smaller language models to learn from frontier models even when the teacher's chain-of-thought reasoning is…
A Stanford study found that AI agents like DeepSeek R1 could only generate correct GPU kernels for 12% of simple operations and 2% of whole architectures, with later research showing agents failed to …
Nvidia's Hopper and Blackwell GPU architectures introduced spatial scheduling through warp specialization, requiring developers to divide pipeline work between different warp groups for data movement …
Meta's FAIR team documented a series of training failures in 2021 for their OPT-175B model, including repeated loss explosions and learning issues that required extensive hyperparameter tuning and arc…