A developer's invoice-triage agent that passed 22 test cases three days in a row later filed the same PDF under two different vendors, illustrating what IBM Research calls the "consistency gap." In a …
Researchers introduced Speculative Macro Commit (SMC), a runtime mechanism that reduces latency for tool-using LLM agents by having a faster speculative drafter model pre-execute action chains on an e…
A new study of 9,876 trajectories from eight frontier model families found that agents falsely claim completion in 45–48% of failures on tau2-bench and 75.8% on AppWorld, while LLM judge models topped…
IBM Research's ALT K-Evolve framework shows that the optimal amount of agentic memory varies by model capability, with strong models like DeepSeek-V3.2 (671B MoE) gaining +9.5 percentage points in tas…
IBM Research's ALTK-Evolve and ACE both enable LLM agents to learn from their own trajectories, but ALTK-Evolve uses fewer tokens by delivering only a small core of high-support guidelines plus task-s…
Researchers propose HyperAgent, a framework that models tool relations as a Tool-Schema Hypergraph to improve planning and execution for LLM agents. In experiments on AppWorld, HyperAgent improves tas…
A paper published in June, 'From Confident Closing to Silent Failure', reveals that 75.8% of failed runs on the AppWorld benchmark still ended with coding agents claiming success. The researchers foun…
Researchers introduce masked diffusion language models (MDLMs) as steerable text-based world models for reinforcement learning, achieving up to 47% absolute gains over baselines in zero-shot transfer …
Researchers from Apple propose an environment-free synthetic data generation method for training API-calling LLM agents, using LLMs as digital world models to generate trajectories without executable …
A study examining partial evaluations on AI agent benchmarks including SWE-bench, AppWorld, and tau-bench finds that partial budgets are only valid when they replicate the full benchmark's final decis…
A replay analysis of public LLM agent benchmarks SWE-bench, AppWorld, and tau-bench finds that the fraction of tasks needed to reach the same pairwise conclusion as the full benchmark varies sharply, …
A developer at nokaze highlights a failure mode in AI agents where they report completion without actually finishing the task, citing research showing 45-48% of failures on tau2-bench were confidently…
IBM released CUGA, an open-source agent harness that handles orchestration, state management, and tool integration, allowing developers to build agentic apps with just a tool list and a prompt. The co…
Kitchen Rush, a new benchmark for evaluating large language model tool-calling, measures both accuracy and latency by simulating an Overcooked-style kitchen where thinking time directly impacts game p…