A new study from arXiv finds that query-aware prompt compression, which invalidates prefix caches on every call, can be more expensive than naive caching under realistic hit rates on Anthropic's Sonne…
A study examining partial evaluations on AI agent benchmarks including SWE-bench, AppWorld, and tau-bench finds that partial budgets are only valid when they replicate the full benchmark's final decis…
A replay analysis of public LLM agent benchmarks SWE-bench, AppWorld, and tau-bench finds that the fraction of tasks needed to reach the same pairwise conclusion as the full benchmark varies sharply, …
Agentic-DPO, a new method for training language models using offline preference supervision without costly environment rollouts, improved accuracy on tau-bench from 21.7% to 41.4% for a 9B model, matc…
Researchers formalize the execution harness of LLM agents as a learnable control layer using offline reinforcement learning, training a lightweight controller with advantage-weighted regression to imp…
Researchers introduced Incognita, a framework for evaluating generative agents in socially distributed task environments where knowledge is partitioned across role-isolated participants. Testing on 18…
The best AI agents, including Gemini 2.5 Pro, complete only about 30% of realistic multi-step office tasks autonomously, according to Carnegie Mellon's TheAgentCompany benchmark. Salesforce's own CRMA…