GitHub Copilot's value lies not in raw model access but in its integrated harness that connects editor, terminal, and PR flow, according to a technical analysis. GitHub's evaluations across SWE-bench …
The KwaiKAT Team at Kuaishou released KAT-Coder-V2.5, an agentic coding model trained on over 100,000 verifiable repository environments spanning 12 languages. The model is available through StreamLak…
Kimi K3 is 30x cheaper than GPT-5 for output tokens while leading the LMArena leaderboard with 1,289 ELO, according to a 2026 comparison. For a customer support bot handling 10M input and 5M output to…
Amazon Web Services released aws-bench, an open-source benchmark for evaluating AI agents on real-world cloud tasks, addressing the need for standardized evaluation as agents become more autonomous. T…
The AI industry is shifting focus from raw novelty to durable interfaces, with the Model Context Protocol locking its largest revision ahead of a July 28 release, AMD unveiling its first 2nm x86 serve…
Vals AI launched Vals-Smith on July 22nd, a tool that converts a GitHub repository's merged pull requests into a private benchmark for testing coding models and agents. The product addresses the probl…
BrassCoders finds that autonomous coding agents, such as Claude Code and SWE-agents, bypass human code review, allowing security vulnerabilities like hardcoded credentials to go undetected. The compan…
A developer ran SWE-bench on a small LLM to map failure modes and understand how tiny models behave under full task-grounded pressure. The experiment validated a multi-stage evaluator pipeline design …
Kimi K3, an open model from Moonshot AI, achieves near-parity with the closed-source Fable 5 on a suite of ~1,030 agentic tasks while costing a fraction of the price, according to a benchmark study by…
A mid-2026 audit of top-down AI-assisted software engineering shows agents absorbing implementation work on schedule, with SWE-bench Verified scores rising from 1.96% in October 2023 to near saturatio…
DeepSWE, a new benchmarking platform for AI coding agents, achieves clearer separation between frontier models than older benchmarks like SWE-bench by using contamination-free tasks across 91 reposito…
Princeton and Stanford researchers released mini-swe-agent, a coding agent in roughly 100 lines of Python that scores over 74% on SWE-bench Verified, beating Claude Code's scaffolding by six percentag…
A benchmark score is a manufactured number that passes through a chain of choices—sampling, prompting, scoring, and aggregation—each of which can change the final result while model weights stay fixed…
AI agent frameworks like Superpowers, Matt Pocock's skills, and Agent Skills use plain English Markdown files as a behavioral control layer, which is probabilistic and ambiguous rather than determinis…
XAI released Grok 4.5 on July 8, trained on trillions of Cursor session tokens capturing real developer-agent interactions, achieving #1 on TAU-bench agentic tool use (71%) and resolving SWE-bench tas…
A developer argues that LLM evaluations for developer tools must measure three distinct axes: usefulness, correctness, and safety. Unlike chatbot evals, dev-tool evals rely on ground-truth signals lik…
A study examining partial evaluations on AI agent benchmarks including SWE-bench, AppWorld, and tau-bench finds that partial budgets are only valid when they replicate the full benchmark's final decis…
A new study from arXiv:2607.12161v1 analyzing 2,848 Claude Code runs across 103 tasks finds that reducing retrieved context or tool output does not reliably lower billed costs for API-based coding age…
A replay analysis of public LLM agent benchmarks SWE-bench, AppWorld, and tau-bench finds that the fraction of tasks needed to reach the same pairwise conclusion as the full benchmark varies sharply, …
Mistral Vibe for Code, Claude Code, Cursor, and OpenAI Codex were scored on a scaffold-to-PR task, with Mistral Vibe and Claude Code tying at 22/25 points. The comparison evaluated five dimensions: fe…