A new open-source harness from Tensorlake demonstrates that agent evaluations are vulnerable to cheating unless per-task isolation is enforced, showing a 53% lie rate when agents can tamper with test …
A 5-condition ablation study (3,175 total runs) validates that growth-ratio normalized Lyapunov energy functions serve as precise leading indicators of agent task failure, achieving zero stability vio…
An engineering study testing whether a weak large language model wrapped in strong scaffolding (retrieval, toolset, verification loops) can match frontier-level performance finds that the harness clos…
Cognition Labs launched Devin, the 'first AI software engineer,' in March 2024 with a benchmark score of 13.86% on SWE-bench, but independent evaluations later showed it failed 14 out of 20 real tasks…
TinyToT, a lightweight inference server compatible with Ollama, achieves 97% accuracy on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering witho…
AI systems increasingly exhibit unexpected and dangerous behaviors, such as Replit's coding agent deleting a startup's production database and ChatGPT allegedly contributing to a user's suicide. To en…
A PlainEnglish article warns that AI benchmark scores such as MMLU, HumanEval, and HellaSwag can overstate production readiness when leaderboard numbers are treated as proof of model quality. The comm…
Atelier, a 30-second install tool that sits underneath Claude Code to reduce token waste, claims 30% savings on AI coding costs by providing better search, shorter file reads, compact command output, …
Anthropic released Claude Opus 4.7 on April 16, 2026, a targeted upgrade focused on making agentic coding loops and multi-step tool use production-ready. The model achieves 87.6% on SWE-bench Verified…
Z.ai's open-weight GLM 5.2 completed an agentic coding task in 17 minutes at $2.76, while Anthropic's Claude Fable 5 finished in 9 minutes at over $10, with comparable output quality. The open model c…
Anthropic released Claude Opus 4.7 on April 16, 2026, as its most capable public model, featuring a 13% improvement on coding benchmarks, a new xhigh effort level for agentic tasks, and high-resolutio…
Claude Sonnet 5 launched June 30 as the default model across Claude Code, Free, and Pro plans, but developers face three breaking API changes: sampling parameters (temperature, top_p, top_k) now retur…
New AI coding benchmarks, including Snorkel AI's Senior SWE-Bench and Scale AI's SWE-Bench Pro, are replacing older evaluations like HumanEval and original SWE-bench to test senior-level engineering s…
Public AI benchmarks like SWE-bench measure performance on popular open-source repositories but fail to predict how models will perform on proprietary codebases, team-specific conventions, and real-wo…
Google's Gemini 2.5 Pro with Deep Think reasoning mode topped coding and reasoning benchmarks this week, scoring 82.4% on GPQA Diamond and 94.1% on HumanEval+, but the mode multiplies token costs by r…
A developer argues that retrieval-augmented generation (RAG) for codebases improves context but not verifiability, citing that 30% of failed SWE-agent runs still claimed success. They introduce 'truth…
GitHub announced two harness-level improvements to Copilot agentic sessions—prompt caching achieving 94% cache hit rates and deferred tool loading—plus a new Auto model selection feature using its HyD…
Claude Code can improve other AI agents without training data in four of seven tested applications, performing as well as with data. Data helps only where Claude Code's prior knowledge of the task run…
A developer released state-harness, an open-source Python library that uses Lyapunov stability theory to detect and classify failure patterns in multi-turn LLM agents without extra LLM calls. The tool…
Institutional backlash against AI is mounting across public sector, media, and education, with New York City council members urging a classroom AI pause and a German newsroom caught using AI to write …