Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

A new arXiv paper, arXiv:2609.11018v1, surveys AI agent evaluation across five dimensions of agenticness — environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence — to address the lack of a standard definition of "agent" in artificial intelligence. The survey synthesizes existing metrics, benchmarks, and evaluation frameworks for each dimension and introduces the Agent Compendium, a public-facing digital resource that organizes and extends the evaluation methods identified in the review. The authors state the work aims to support more reproducible research, clearer communication, and more systematic study of artificial agents.

arXiv:2609.11018v1 Announce Type: new Abstract: The term agent in artificial intelligence lacks a standard definition, complicating the evaluation, comparison, and reproducibility of AI agent research. We address this ambiguity through a survey organized around five dimensions of agenticness: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. For each dimension, we examine how the underlying capability has been conceptualized across prior work and synthesize the metrics, benchmarks, and evaluation frameworks used to assess it. This review provides a structured account of the current landscape of agent evaluation, highlighting both established approaches and areas where evaluation remains limited or inconsistent. We additionally introduce the Agent Compendium, a public-facing digital resource that organizes and extends the evaluation methods identified through this review. Together, the survey and compendium provide a common structure for evaluating and comparing agent capabilities across AI systems, supporting more reproducible research, clearer communication, and more systematic study of artificial agents.