A developer's context-selection tool, cognitive-cache, initially appeared no better than grep in a small benchmark, but a larger SWE-bench evaluation showed it significantly outperformed grep and TF-I…
A developer reports that the gap between AI models and their agentic performance is often determined by the harness—the system around the model—rather than the model itself. Evidence from Moonshot's K…
Prime Intellect has integrated 23 agentic tasksets into a unified API, providing approximately 365,000 tasks for software engineering, terminal, and search domains, ready for reinforcement learning tr…
NVIDIA open-sourced NOOA, an AI agent framework that treats agents as Python objects, achieving 82.2% on SWE-bench Verified with GPT-5.5 at roughly half the token cost of comparison systems. The frame…
GitHub Copilot's value lies not in raw model access but in its integrated harness that connects editor, terminal, and PR flow, according to a technical analysis. GitHub's evaluations across SWE-bench …
The KwaiKAT Team at Kuaishou released KAT-Coder-V2.5, an agentic coding model trained on over 100,000 verifiable repository environments spanning 12 languages. The model is available through StreamLak…
Kimi K3 is 30x cheaper than GPT-5 for output tokens while leading the LMArena leaderboard with 1,289 ELO, according to a 2026 comparison. For a customer support bot handling 10M input and 5M output to…
Amazon Web Services released aws-bench, an open-source benchmark for evaluating AI agents on real-world cloud tasks, addressing the need for standardized evaluation as agents become more autonomous. T…
The AI industry is shifting focus from raw novelty to durable interfaces, with the Model Context Protocol locking its largest revision ahead of a July 28 release, AMD unveiling its first 2nm x86 serve…
Vals AI launched Vals-Smith on July 22nd, a tool that converts a GitHub repository's merged pull requests into a private benchmark for testing coding models and agents. The product addresses the probl…
BrassCoders finds that autonomous coding agents, such as Claude Code and SWE-agents, bypass human code review, allowing security vulnerabilities like hardcoded credentials to go undetected. The compan…
A developer ran SWE-bench on a small LLM to map failure modes and understand how tiny models behave under full task-grounded pressure. The experiment validated a multi-stage evaluator pipeline design …
Kimi K3, an open model from Moonshot AI, achieves near-parity with the closed-source Fable 5 on a suite of ~1,030 agentic tasks while costing a fraction of the price, according to a benchmark study by…
A mid-2026 audit of top-down AI-assisted software engineering shows agents absorbing implementation work on schedule, with SWE-bench Verified scores rising from 1.96% in October 2023 to near saturatio…
DeepSWE, a new benchmarking platform for AI coding agents, achieves clearer separation between frontier models than older benchmarks like SWE-bench by using contamination-free tasks across 91 reposito…
Princeton and Stanford researchers released mini-swe-agent, a coding agent in roughly 100 lines of Python that scores over 74% on SWE-bench Verified, beating Claude Code's scaffolding by six percentag…
A benchmark score is a manufactured number that passes through a chain of choices—sampling, prompting, scoring, and aggregation—each of which can change the final result while model weights stay fixed…
AI agent frameworks like Superpowers, Matt Pocock's skills, and Agent Skills use plain English Markdown files as a behavioral control layer, which is probabilistic and ambiguous rather than determinis…
XAI released Grok 4.5 on July 8, trained on trillions of Cursor session tokens capturing real developer-agent interactions, achieving #1 on TAU-bench agentic tool use (71%) and resolving SWE-bench tas…
A developer argues that LLM evaluations for developer tools must measure three distinct axes: usefulness, correctness, and safety. Unlike chatbot evals, dev-tool evals rely on ground-truth signals lik…