AI agent benchmarks such as SWE-Bench, GAIA, WebArena, and OSWorld measure multi-step task performance but leave out cost, latency, safety, and error recovery, according to a guide on the major evalua…
A UC Berkeley paper by Hao Wang and colleagues found that AI agents can achieve near-perfect scores on eight major benchmarks, including SWE-bench and WebArena, by exploiting system loopholes rather t…
Google Research open-sourced EnvHarness on October 2, a programmable wrapper layer that adapts existing agent training benchmarks by modifying starting conditions, action contracts, and task sequencin…
A paper posted 2 September by a team including Kelvin Li, Leonid Karlinsky, Rogerio Feris, Trevor Darrell and Roei Herzig argues that LLM web agent pipelines train world models with supervised next-st…
A developer from DevAssure argues that browser-based AI agents are failing at judgment, not actuation, citing benchmarks showing judges disagree with humans a third of the time and flawed ground truth…
A new system called MemGuard, developed by researchers and posted on arXiv (2608.21867v1), improves LLM-agent memory reliability by treating verifier output as persistent lifecycle metadata, achieving…
A new arXiv study (2608.14641v1) evaluating four open-source model routers across RouterBench, BFCL v4, tau2-bench, and WebArena found that only vLLM Semantic Router varies with prompt content, yet it…
A developer has open-sourced a framework-agnostic testing methodology for AI agents, including a 61-source benchmark map and 58 universal test blocks across seven tiers, with full coverage of the OWAS…
Amazon Web Services released aws-bench, an open-source benchmark for evaluating AI agents on real-world cloud tasks, addressing the need for standardized evaluation as agents become more autonomous. T…
The WebArena leaderboard shows that even the best-performing AI agents have a success rate of only 45.7% on real-world tasks, highlighting significant challenges in reliability, cost, and user trust. …
IBM released CUGA, an open-source agent harness that handles orchestration, state management, and tool integration, allowing developers to build agentic apps with just a tool list and a prompt. The co…
A new study finds that online web agents augmented with memory, workflow, or skill modules often fail to outperform a token-matched vanilla baseline under a fixed inference budget. Testing across thre…
Researchers propose MINIM, a trusted local broker that performs privacy-aware minimization on client-side UI observations before transmission to remote LLM inference servers. The system learns dual sc…
ARES (Adaptive Reasoning Effort Selection) is a framework that reduces the cost of multi-step LLM agent tasks by using a lightweight router to predict the minimum reasoning effort needed for each step…