A new analysis argues that claims about large language models having preferences, such as 'Claude prefers React,' are misleading because models lack preferences and instead reflect training data and c…
AI evaluations are failing as models approach general intelligence, with benchmarks saturating through contamination and Goodhart effects while the scope of evaluation expands from minutes to months. …
A data scientist reduced monthly AI inference costs from $3,200 to $580 by switching from GPT-4 to Chinese models like DeepSeek V4 Flash and DeepSeek R1. The switch leveraged OpenAI-compatible APIs fr…
Researchers at the Allen Institute for AI used training-data attribution to map which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in the OLMo3-7B language model. T…
Arcadia Alignment's research reveals that current AI model organisms used to study alignment pathologies suffer from degraded coherence, instruction-following, and reasoning, making them poor proxies …
A new guide explains how to build a personal AI model leaderboard by running blind comparisons and tracking results over time, arguing that public benchmarks are insufficient for task-specific perform…
LLM benchmarks like MMLU and HumanEval are irrelevant for most businesses building AI products, as they measure generic performance rather than specific system tasks. Teams should instead build custom…
Sapient Inc. open-sourced HRM-Text in May 2026, a 1.15 billion parameter text generation model based on the HRM architecture. Trained on roughly 40 billion tokens—up to 1,000 times less data than comp…
Andon Labs co-founders Lukas Petersson and Axel Backlund are stress-testing frontier AI models by giving them real-world control over inventory, wallets, and physical storefronts, revealing unexpected…
Anthropic's Claude Opus 4.8 autonomously built a full economic simulation benchmark from scratch, designing test scenarios, generating ground-truth outcomes, and writing evaluation rubrics without hum…
A new paired-minimum detectable effect (MDE) budget for 4-bit quantization benchmarks, derived from classical sample-size calculations, provides benchmark designers a one-line formula to pre-register …
A new study from arXiv (2605.29027) found that tonal variations in prompts cause systematic but model-dependent accuracy shifts in large language models (LLMs) on objective multiple-choice questions. …
Researchers at OpenPangu identified two failure modes in HiF8 W8A8 quantization-aware training for the OpenPangu-Embedded-1B model: amax saturation, which corrupts knowledge-sensitive representations …
A developer argues that current LLM benchmarks like MMLU, HumanEval, and SWE-bench measure only knowledge recall and one-shot task completion, not the behavioral traits—such as debugging, adaptation, …
A new analysis of over 5,400 AI models reveals that benchmark scores for large language models are highly correlated, with just five subjects on the MMLU test predicting the remaining 52 with 91% accu…
A new independent certification system for AI agents, where each candidate is evaluated in parallel by four different AI models from four different providers (Anthropic, Cerebras, Moonshot, and Groq).…