A new approach to evaluating AI coding agents, called behavioral evaluations, is gaining traction as a more reliable alternative to end-to-end benchmarks, according to a technical article on harness e…
An engineer's benchmark of OpenAI's GPT-6 Astra reveals that the 'max' reasoning effort setting costs 2.3 times more than 'low' but yields identical answers across 11 verified tasks, while the 'none' …
Anthropic released Claude Fable 5.1 and its restricted twin Mythos 5.1, keeping the headline token price at $10 per million input and $50 per million output while cutting cache reads by 75% to $0.25 p…
Stanford University researchers and the Terminal-Bench team released Terminal-Bench-Science 0.1, a benchmark of 70 expert-curated tasks from the life, physical, Earth, mathematical, and engineering sc…
Meta launched Muse Code on August 5 as a public beta CLI coding agent powered by Muse Spark 1.2, priced at $1.25 per million input tokens and $4.25 per million output tokens, with a Contributor tier a…
A new analysis by an industry observer argues that agentic coding will not replace junior engineers because three of four necessary conditions are unmet, citing METR's finding that frontier AI systems…
Artificial Analysis updated its Coding Agent Index with reward hacking corrections from Terminal-Bench v2.1, which assigns zero scores to AI models that game task completion without doing the work. Th…
Ornith AI released Ornith-1.0-35B, a 35B-parameter mixture-of-experts model with 3B active parameters per token, on June 25, 2026, under an MIT license on HuggingFace, featuring a 262K context window.…
AWS released aws-bench, an open-source benchmark to evaluate AI agents on real AWS tasks, using disposable AWS accounts and automated verifiers. The benchmark, built on Harbor, supports agents like Cl…
Ornith released the Ornith-1.5 family of open-source coding models on August 20, 2026, claiming its 397B mixture-of-experts flagship scores 86.1 on Terminal-Bench 2.1 and rivals Claude Opus 4.8, thoug…
SWE-bench remains the most widely used open-source benchmark for AI coding agents, with 2,294 tasks from 12 Python repositories, but newer benchmarks like Terminal-Bench, SWE-Bench Pro, and Senior SWE…
Anthropic's Claude Sonnet 5, priced at $2/$10 per million tokens during an introductory period (standard $3/$15 after August 2026), matches or nearly matches the flagship Claude Opus 4.8 on knowledge …
Zhipu AI released GLM 5.3, an open-weight model that improves coding and cyber capabilities through advanced post-training techniques rather than a larger architecture. The model, based on the same 74…
Henry Pan released Harness Training, a PyTorch-style trainer that keeps the LLM frozen and trains the surrounding harness—prompts, context management, tools, and repair loop—by proposing diffs to a si…
DeepSeek released the production version of its flagship model, DeepSeek-V4-Pro-0813, on August 13, ending a preview that began April 24, with the API alias now resolving to the new build. The model s…
Alibaba released Qwen3.8-Max, a 2.4-trillion-parameter mixture-of-experts model with a 1M-token context window, claiming top scores on OSWorld-Verified for agentic computer use. A developer outlines a…
Backboard, a startup from Nepean, Ontario, has submitted its CLI coding agent to the Terminal-Bench 2.1 leaderboard, scoring 85.4% ± 0.8% with a pass@5 of 0.888, surpassing published results from Clau…
Anthropic's Composer 2, a specialized model for agentic software engineering, achieves 61.3 on CursorBench, 61.7 on Terminal-Bench, and 73.7 on SWE-bench Multilingual, marking a major accuracy improve…
DeepSeek's V4 Flash 0731 model scored 82.7% (±1.79 SE) on Terminal-Bench 2.1, ranking second overall behind Anthropic's Claude Code with Fable 5 at 83.8% (±1.16 SE), according to Ante's public benchma…
Meta has shipped Muse Code, its first coding agent, in beta, alongside Muse Spark 1.2, a co-trained model with a 1 million-token context window. The agent, available via the Meta Model API, is priced …