A PlainEnglish article warns that AI benchmark scores such as MMLU, HumanEval, and HellaSwag can overstate production readiness when leaderboard numbers are treated as proof of model quality. The comm…
A new study finds that prompt robustness in large language models depends on the type of question asked, with objective questions showing different sensitivity to prompt variations than subjective bel…
Researchers introduce PuzzleMoE, a method for compressing large Mixture-of-Experts models via fine-grained element-wise merging and bit-packing, achieving up to 16.7% higher accuracy on MMLU at 50% co…
The Center for AI Safety, with world experts, created Humanity's Last Exam (HLE), a benchmark of over 2,500 expert-level questions across disciplines to test AI reasoning. Even top models like GPT, Ge…
A reinforcement learning-driven data scheduler, AC-ODM, boosts MMLU performance by 27.5% relative and HumanEval pass@1 by 2.23× on a Pythia-1B model with only a 0.4% per-step wall-clock increase and 2…
A developer recounts pushing an agent into production that failed after a CRM dropdown change, highlighting the inadequacy of model-level evaluation for agent systems. The article argues that agents m…
Hugging Face and the EvalEval Coalition launched an integration that allows contributors to submit standardized evaluation results (EEE schema) to Hugging Face Community Evals, consolidating scattered…
Recent AI research introduces RL-driven agentic optimization using dense token-level supervision and progress advantage signals to stabilize training. PhysiFormer injects 3D geometric reasoning into d…
A viral chart predicting open-weights AI models will catch closed frontier models by December 2026 is misleading, as analysis across 18 benchmarks shows the gap varies by task and is not uniformly shr…
A developer observed that after multiple context compactions in LLM sessions, output quality degrades non-linearly, with a brief improvement after the second compaction before declining. They built a …
Researchers proposed Dynamic-dLLM, a training-free framework to accelerate Diffusion Large Language Models (dLLMs) by dynamically allocating cache budgets and calibrating decoding thresholds. The meth…
A practical guide for benchmarking fine-tuned models recommends starting with a held-out test set matching the actual task rather than relying solely on public benchmarks. The workflow includes defini…
Microsoft Principal Developer Advocate Waldek Mastykarz published a blog post arguing that testing large language models in empty chat sessions is methodologically flawed, as models have no preference…
A new analysis argues that claims about large language models having preferences, such as 'Claude prefers React,' are misleading because models lack preferences and instead reflect training data and c…
AI evaluations are failing as models approach general intelligence, with benchmarks saturating through contamination and Goodhart effects while the scope of evaluation expands from minutes to months. …
A data scientist reduced monthly AI inference costs from $3,200 to $580 by switching from GPT-4 to Chinese models like DeepSeek V4 Flash and DeepSeek R1. The switch leveraged OpenAI-compatible APIs fr…
Researchers at the Allen Institute for AI used training-data attribution to map which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in the OLMo3-7B language model. T…
Arcadia Alignment's research reveals that current AI model organisms used to study alignment pathologies suffer from degraded coherence, instruction-following, and reasoning, making them poor proxies …
A new guide explains how to build a personal AI model leaderboard by running blind comparisons and tracking results over time, arguing that public benchmarks are insufficient for task-specific perform…
LLM benchmarks like MMLU and HumanEval are irrelevant for most businesses building AI products, as they measure generic performance rather than specific system tasks. Teams should instead build custom…