A new guide explains how to build a personal AI model leaderboard by running blind comparisons and tracking results over time, arguing that public benchmarks are insufficient for task-specific perform…
LLM benchmarks like MMLU and HumanEval are irrelevant for most businesses building AI products, as they measure generic performance rather than specific system tasks. Teams should instead build custom…
Sapient Inc. open-sourced HRM-Text in May 2026, a 1.15 billion parameter text generation model based on the HRM architecture. Trained on roughly 40 billion tokens—up to 1,000 times less data than comp…
Andon Labs co-founders Lukas Petersson and Axel Backlund are stress-testing frontier AI models by giving them real-world control over inventory, wallets, and physical storefronts, revealing unexpected…
Anthropic's Claude Opus 4.8 autonomously built a full economic simulation benchmark from scratch, designing test scenarios, generating ground-truth outcomes, and writing evaluation rubrics without hum…
A new paired-minimum detectable effect (MDE) budget for 4-bit quantization benchmarks, derived from classical sample-size calculations, provides benchmark designers a one-line formula to pre-register …
A new study from arXiv (2605.29027) found that tonal variations in prompts cause systematic but model-dependent accuracy shifts in large language models (LLMs) on objective multiple-choice questions. …
Researchers at OpenPangu identified two failure modes in HiF8 W8A8 quantization-aware training for the OpenPangu-Embedded-1B model: amax saturation, which corrupts knowledge-sensitive representations …
A developer argues that current LLM benchmarks like MMLU, HumanEval, and SWE-bench measure only knowledge recall and one-shot task completion, not the behavioral traits—such as debugging, adaptation, …
A new analysis of over 5,400 AI models reveals that benchmark scores for large language models are highly correlated, with just five subjects on the MMLU test predicting the remaining 52 with 91% accu…
A new independent certification system for AI agents, where each candidate is evaluated in parallel by four different AI models from four different providers (Anthropic, Cerebras, Moonshot, and Groq).…