Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility

Researchers introduced tool efficiency and marginal tool utility, new quantitative metrics to evaluate the usefulness of tool calls in LLM agent trajectories, using LLM-as-a-Judge to determine whether each tool call is beneficial. The work aims to directly measure efficiency in post hoc analyses, complementing accuracy-based evaluations and informing future benchmark designs and lean tool suite engineering.

arXiv:2607.14108v1 Announce Type: new Abstract: This paper introduces tool efficiency, a new quantitative metric to evaluate the rate of useful tool calls in an LLM agent trajectory. To ensure that tool efficiency is well-defined, we also introduce marginal tool utility, a new quantitative metric defined per tool call indicating whether a tool is useful or whether it can be safely removed from the tool suite without affecting accuracy while increasing tool efficiency; in this paper, we determine the sign of marginal tool utility for each tool call in a trajectory using LLM-as-a-Judge. While much prior work has been done to develop techniques that improve tool use by LLMs and design evaluation methods measuring efficiency indirectly using accuracy as a proxy, our work is centered on measuring efficiency directly via the quantitative metric proposed in this paper in post hoc trajectory analyses. It is our intention that this work contributes to the frontier of LLM evaluation research as a springboard for future benchmark designs and agent harness engineering specifically with regards to creating lean tool suites that optimize for metrics that complement but are distinct from accuracy.