OpenAI reviewed the SWE-Bench Pro AI coding test and found roughly 30% of its tasks are flawed, leading the company to withdraw its endorsement. The problems include overly strict or vague task requir…
An evaluation of Anthropic's Sonnet 5 and Opus 4.8 models across 24 coding tasks shows Sonnet produces clearer and more intentional patches, while Opus yields simpler, more robust, and more minimal di…
OpenAI published research on using AI to help diagnose rare genetic diseases in children, emphasizing the model's role in narrowing diagnostic search space rather than replacing physicians. Anthropic …
DeepSWE v1.1 updates the benchmark for long-horizon engineering tasks with isolated verification and structured test reports, making results more reproducible and harder to game. Pass rates remain clo…
Fable 5 has achieved a performance level on par with GPT-5.5 in the Artificial Analysis Coding Agent Index, a composite benchmark measuring real-world coding agent performance across software engineer…
Researchers have found that Claude Opus was contaminated on approximately 12% of SWEBench Pro tasks, meaning the model may have encountered those benchmark problems in its training data. The contamina…
The cost of frontier proprietary models increased significantly in May 2026, with Gemini 3.5 Flash priced three times higher than its predecessor and GPT 5.5 costing double GPT 5.4, while open-weight …
A DeepSWE benchmark audit found that DeepSeek-v4-pro's reported 8% solve rate and $4.22 average cost are unreliable due to inflated pricing and procedural errors. The audit team solved all three tasks…
DeepSWE has demonstrated that a fully optimized GPT 5.5 model can generate more intelligence at a lower cost than a maxed-out Opus 4.8, marking a significant efficiency breakthrough in AI performance.…
Anthropic released Claude Opus 4.8, an incremental upgrade with improved agentic judgment and honesty, achieving 84% on Online-Mind2Web and roughly 4x fewer code flaw overlooks. Harvard's Zitnik Lab i…
OpenAI has acknowledged in its official documentation that its GPT-5.5 model exhibits "diminished intelligence" and reduced nuance in complex creative generation compared to its predecessor, validatin…
DataCurve released DeepSWE, a new benchmark for evaluating frontier coding agents on original, long-horizon software engineering tasks. The benchmark features contamination-free tasks written from scr…