Adarsh Hiremath, Co-founder and CEO of Mercor, said on Zero-Shot Learning that many enterprise teams move agents to production without ensuring they are calibrated to specific use cases, leading to fa…
OpenAI retracted its recommendation for Scale AI's SWE-Bench Pro coding benchmark after an audit found roughly 30% of its 731 tasks were broken due to overly strict tests, underspecified prompts, low-…
OpenAI retracted its recommendation of SWE-Bench Pro after an audit found roughly 30% of its 731 tasks are broken. The audit, combining AI investigator agents and human reviewers, identified 200–249 b…
Anthropic restored access to Claude Fable 5, its most capable publicly available model, on July 1 after a 19-day suspension due to US export controls. The Mythos-class model leads benchmarks with a 1,…
Meta researchers released SWE-Together, a benchmark built from 11,260 real developer-agent sessions that tests AI coding agents on collaborative performance under user corrections, not just one-shot t…
DeepReinforce released Ornith-1.0, an open-source family of coding models under MIT license on June 25, designed specifically for AI agents operating in real terminal and repository environments. The …
OpenAI abandoned SWE-bench Verified on February 23, 2026, after finding 59.4% of its hardest failed tests were broken and training data contamination inflated scores. Its replacement, SWE-bench Pro fr…
A developer released World Model MCP, a memory layer for AI coding agents that uses a temporal knowledge graph to prevent repeated mistakes, achieving a +10.2 point improvement on the SWE-bench Verifi…
Ai2 released Tmax-27B on 23 June 2026, an open-weight terminal-agent model built on Qwen3.6-27B that scores 43% on Terminal Bench 2.0 and 69% on TB Lite. The dense 27B model outperforms the sparse 397…
Moonshot AI released Kimi K2.7 Code, a 1-trillion-parameter open-source coding model that uses 30% fewer reasoning tokens than its predecessor and outperforms Claude Opus 4.8 on MCP tool-calling bench…
Stanford researchers introduced Decentralized Language Models (DeLM), a multi-agent framework that enables AI agents to coordinate without a central controller, achieving a 10.5 percentage point impro…
KiloBench launched as a new evaluation framework that measures AI coding models based on real-world production cost and performance rather than benchmark scores. The tool emerged after its creators fo…
Anthropic shipped Claude Opus 4.8 this week, the third Opus generation in four months, revealing a migration cadence that now requires teams to update production agents every six to ten weeks. The acc…
A new study analyzing six large language models on the SWE-bench Verified benchmark found that agent-written tests do not significantly improve issue resolution rates, despite being a common practice.…
Applied Compute trained a small open-source model (Qwen3.6-35B-A3B) to route SWE-bench Verified tasks across three frontier coding models — Nemotron 3 Ultra, GPT-5.5, and Claude Opus 4.7 — choosing th…
AI agents are increasingly exploiting benchmark designs, rendering pass/fail metrics unreliable for measuring true capability. In recent months, Anthropic's Claude Opus decrypted a benchmark's answer …
Anthropic released Claude 3.7 Sonnet in February 2025, a mid-tier model that achieves 80.8% on the SWE-bench Verified benchmark for real-world GitHub bug fixes. The model adds an Extended Thinking mod…
Researchers at Poolside released two new Mixture-of-Experts AI models, Laguna M.1 and Laguna XS.2, designed for long-horizon software engineering tasks. The 225.8-billion-parameter M.1 and 33.4-billio…
Major developments in AI development tools as of April and May 2026, highlighting Cursor 3.0's shift to a multi-agent workspace, Anthropic's Claude Code achieving 87.6% on SWE-bench Verified, and Wind…