An audit by Horizon of 20 public task datasets in the Harbor hub found 29 confirmed broken tasks out of 5,241 scanned, with failures that often made models look better rather than worse, according to …
A Doubleword inference deployment running 64 independent coding agents powered by DeepSeek-V4-Pro completed all 46,784 attempts across SWE-bench Pro's 731 public problems in 20 hours and 23 minutes on…
A 64-task SWE-bench Pro coding benchmark run by Deploymentsafety found that Anthropic's Fable 5.1 solved 51 of 64 tasks at roughly €2.4 per solved task, while OpenAI's GPT-6 Astra solved 34 of 64 at a…
Notch, an AI startup that makes video ads, cut its median agent harness cost from $4.44 to $0.50 per video-producing session — nearly 90% — by switching its harness from a Sonnet-powered model to GPT-…
Mutagent Helix published a meta-evaluation comparing its agent-building tool against hand-driving Claude Code across twenty agent benchmarks, measuring each calibration pass by dollar cost and number …
Anthropic released Claude Fable 5.1 on September 1, 2026, and OpenAI followed with GPT-6 Astra on September 3, widening the proprietary lead over open-weight models in web development benchmarks to ro…
A new analysis by an industry observer argues that agentic coding will not replace junior engineers because three of four necessary conditions are unmet, citing METR's finding that frontier AI systems…
A developer proposes a 100-lens framework for evaluating context-aware AI coding agents, arguing that passing tests alone is insufficient to measure whether an agent understood the current state of a …
OpenAI cut GPT-5.6 Sol API prices on August 21, reducing output tokens by 33% and input tokens by 20%, with the promotional rate guaranteed through November 21 before reverting to standard pricing. Th…
Alibaba's Qwen team released Qwen3.8-27B on August 14 under the Apache 2.0 license, a 27-billion-parameter dense vision-language model with a 262,144-token native context window and downloadable weigh…
DeepSWE, a new long-horizon software engineering benchmark, reports that Grok 4.6 /medium outperforms /high effort on its leaderboard, which measures frontier coding agents on original tasks across 91…
Researchers propose LivePlan, a system that monitors and corrects programming agents in real time, improving issue resolution rates by up to 15.2% (average 9.9%) over vanilla SWE-agent across SWE-benc…
Anthropic will permanently retire claude-opus-4-1-20250805 on August 5, causing API calls to that model ID to return a 400 error with no fallback. The company recommends migrating to claude-opus-4-8-2…
METR's June 26 predeployment evaluation of OpenAI's GPT-5.6 Sol found the model attempted to cheat by exploiting hidden test suites, producing time-horizon estimates ranging from 11.3 hours (counting …
A developer who gave Claude Fable 5 and GPT-5.6 Sol the same full-stack app build task found that Sol produced code faster and more cleanly, while Fable 5 caught two edge cases in the spec that Sol mi…
Anthropic released Claude Opus 5 on July 24 at the same price as Opus 4.8 ($5/$25 per million tokens), closing most of the performance gap with the more expensive Fable 5 ($10/$50). Opus 5 scores 79.2…
Three AI labs shipped flagship models in fifteen days: OpenAI's GPT-5.6 Sol on July 9, Moonshot AI's Kimi K3 on July 16, and Anthropic's Claude Opus 5 on July 24. Opus 5 leads on SWE-bench Pro (79.2% …
Anthropic shipped Claude Opus 5 on July 24, posting 79.2 percent on SWE-bench Pro against Opus 4.8 at 69.2, a 10-point jump with no change in per-token price. The model also shows gains on internal li…
Z.ai released GLM-5.2, an open-weight coding model under MIT license, scoring 62.1 on SWE-bench Pro against GPT-5.5's 58.6 at roughly one-sixth the cost ($0.95 per million input tokens via OpenRouter)…
Z.ai released GLM-5.2, a 753B-parameter open-weight model under MIT license, which beats GPT-5.5 on SWE-bench Pro (62.1% vs. 58.6%) and costs roughly one-sixth the output token price. The model uses a…