The developers behind the open-source OmnisBench project have released a benchmark designed to measure the true cost savings of LLM routing. Running 364 coding and math tasks across four models, they …
A vendor-agnostic framework from DORA and other sources shows that AI coding tools should be judged by delivery outcomes such as change failure rate and time to restore service, not by vendor benchmar…
A developer's guide outlines the three pillars for building production-ready AI agents: benchmarking, cost optimization, and tooling. It emphasizes task-level evaluation over static benchmarks, using …
A new arXiv preprint (2608.12599v1) introduces a method to measure and restore constraint influence in black-box LLM dialogues, finding that behavioral relapse—models continuing to enact revoked const…
Researchers propose CORA-Diff, a training-free method that uses native confidence and persistence signals to skip unnecessary denoising steps in diffusion language models, achieving up to 3.32x speedu…
A developer built a 30-minute harness that tests coding models against five tasks mined from their own git history, scoring results with their real test suite. The harness, which runs any coding model…
A new arXiv paper (arXiv:2608.10333v1) introduces MERA, a method that improves small language models through multi-cycle adaptation, raising Qwen2.5-Coder-1.5B from 28.7% to 49.7% pass on held-out Hum…
An agent that passes 75 out of 100 test runs has a 75% pass@1 rate, but reliability depends on the product: pass@k (at least one success in k tries) yields 98.4% for k=3, while pass^k (all k succeed) …
A new tutorial from Anthropic demonstrates that AI agents can reliably self-correct only when grounded in external verification, citing a 2024 paper showing large language models cannot self-correct r…
DeepSeek V3 generally outperforms Claude 3.5 Sonnet on raw coding benchmarks like HumanEval and MBPP, especially in C++, Rust, and mathematical implementations, while Claude 3.5 Sonnet is considered s…
A developer argues that LLM evaluations for developer tools must measure three distinct axes: usefulness, correctness, and safety. Unlike chatbot evals, dev-tool evals rely on ground-truth signals lik…
A new study reveals that the multiplicative repetition penalty used across major LLM inference engines (HuggingFace, vLLM, llama.cpp) is fundamentally flawed because it branches on the sign of raw log…
XAI's Grok 4.5 scored 83% on Terminal Bench, outperforming many costlier models in multi-step coding tasks, while Anthropic's Claude Opus 4.8 excels in complex reasoning and high-stakes professional w…
A PlainEnglish article warns that AI benchmark scores such as MMLU, HumanEval, and HellaSwag can overstate production readiness when leaderboard numbers are treated as proof of model quality. The comm…
NVIDIA open-sourced Nemotron-Labs-TwoTower, a diffusion language model that generates text 2.42x faster than its autoregressive counterpart without retraining original weights. The model achieves 98.7…
Researchers propose Trajectory-Aware Commit Gating (TACG), a training-free decoder for diffusion language models that uses trajectory-aware signals to decide when to commit tokens, improving accuracy …
New AI coding benchmarks, including Snorkel AI's Senior SWE-Bench and Scale AI's SWE-Bench Pro, are replacing older evaluations like HumanEval and original SWE-bench to test senior-level engineering s…
A reinforcement learning-driven data scheduler, AC-ODM, boosts MMLU performance by 27.5% relative and HumanEval pass@1 by 2.23× on a Pythia-1B model with only a 0.4% per-step wall-clock increase and 2…
Google's Gemini 2.5 Pro with Deep Think reasoning mode topped coding and reasoning benchmarks this week, scoring 82.4% on GPQA Diamond and 94.1% on HumanEval+, but the mode multiplies token costs by r…
Nearly half of 60 popular AI benchmarks studied in a new ICML paper show high levels of saturation, meaning they can no longer reliably distinguish between leading models. The researchers from the pap…