Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

Researchers argue that aggregate-score leaderboards for LLM agent benchmarks systematically underspecify deployed-agent evaluation, as rankings do not transfer to out-of-distribution settings. They propose ranking configurations by predictive validity—correlation between in-sample and out-of-sample rank—and present a twelve-tier measurement apparatus to expose deployment-relevant dimensions. The paper closes with a pre-registered pilot design and a vision for next-generation agentic benchmarks.

arXiv:2606.19704v1 Announce Type: new Abstract: Agent benchmarks are growing fast, but no single benchmark touches more than four or five of the dimensions that deployment exposes. This paper aggregates the largest coordinated deep-dive of one MCP-based industrial-agent benchmark to date: fourteen parallel implementation studies covering new asset classes including a multi-modal visual extension , alternative orchestrations, retrieval strategies, reasoning modes, infrastructure optimizations, and evaluation-methodology probes. Consolidating those studies with seven prior agent benchmarks, we argue that aggregate-score leaderboards systematically underspecify deployed-agent evaluation. Rankings derived from aggregate scores do not transfer to out-of-distribution settings; recent public-to-hidden competition retrospectives provide direct empirical evidence of this rank instability. We propose ranking configurations by predictive validity, the correlation between in-sample and out-of-sample rank, rather than in-sample mean, and report a twelve-tier measurement apparatus that exposes the deployment-relevant dimensions HELM and its agent-era successors collapse. The position is operationalized through three falsifiable out-of-distribution criteria with explicit thresholds; existing evidence partly supports it but is too thin to confirm. We close with a pre-registered pilot design and a field-level vision for what the next generation of agentic benchmarks should report.