{"slug": "how-many-tasks-are-enough-for-agent-benchmark-decisions-a-replay-analysis-of-llm", "title": "How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks", "summary": "A replay analysis of public LLM agent benchmarks SWE-bench, AppWorld, and tau-bench reveals that the fraction of tasks needed to reach the same pairwise conclusion as the full benchmark varies sharply, from 15% for AppWorld to 90% for SWE-bench Verified, with SWE-bench Lite failing to meet targets even at 95%. The study highlights that partial evaluations must specify performance thresholds, task selection, coverage and decision rules, and unresolved comparison limits.", "body_md": "arXiv:2607.12338v1 Announce Type: new\nAbstract: Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark. We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench. A partial budget counts as enough only when it supports the completed benchmark's decision, covers required task groups, and leaves no more than a target fraction of comparisons unresolved. The required task fraction varies sharply. At the strict 0 percentage point threshold on a 5 percentage point budget grid, AppWorld first meets all targets at 15 percent, tau-bench at 25 percent, and SWE-bench Verified at 90 percent; SWE-bench Lite does not meet all targets by 95 percent under the primary coverage rule. Partial-evaluation reports should state how much one agent must outperform another, how tasks are selected, what coverage rule is required, what decision rule is used, and how many comparisons may remain unresolved.", "url": "https://wpnews.pro/news/how-many-tasks-are-enough-for-agent-benchmark-decisions-a-replay-analysis-of-llm", "canonical_source": "https://arxiv.org/abs/2607.12338", "published_at": "2026-07-15 04:00:00+00:00", "updated_at": "2026-07-15 04:23:12.322100+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents"], "entities": ["SWE-bench", "AppWorld", "tau-bench"], "alternates": {"html": "https://wpnews.pro/news/how-many-tasks-are-enough-for-agent-benchmark-decisions-a-replay-analysis-of-llm", "markdown": "https://wpnews.pro/news/how-many-tasks-are-enough-for-agent-benchmark-decisions-a-replay-analysis-of-llm.md", "text": "https://wpnews.pro/news/how-many-tasks-are-enough-for-agent-benchmark-decisions-a-replay-analysis-of-llm.txt", "jsonld": "https://wpnews.pro/news/how-many-tasks-are-enough-for-agent-benchmark-decisions-a-replay-analysis-of-llm.jsonld"}}