Agent evaluation metrics and AI agent benchmarks: what each one measures

A reference published in November 2026 defines agent evaluation metrics with formulas and maps them to the AI agent benchmarks that report them, grouping the metrics into outcome, trajectory, efficiency and safety categories. The reference states that identical metric names can diverge across implementations, noting that two popular eval libraries can return 0 and 100% for the same tool-call trace, and that benchmark task success and dashboard task success can differ in grader, number of attempts and step limit. The benchmark table was read from each benchmark's own site or paper on November 13, 2026, and the author discloses building EvalShift, an eval tool that computes some of the metrics.

← all posts https://www.evalshift.dev/blog Agent evaluation metrics and AI agent benchmarks: what each one measures Agent evaluation metrics defined with formulas, the AI agent benchmarks that report them as of November 2026, and which ones you can compute yourself. Agent evaluation metrics are the numbers that describe how an AI agent behaved on a task: whether it finished, which tools it called and with what arguments, whether it broke a rule, how many steps and dollars it used, and how often it does the same thing twice. They fall into four groups: outcome, trajectory, efficiency and safety. An AI agent benchmark is a fixed task set that reports one or two of them for a model, usually an average success rate. "Task success" on a benchmark and "task success" on your dashboard can share a name and little else: a different grader, a different number of attempts, a different step limit. The same goes for tool-call accuracy, where two popular eval libraries can return 0 and 100% for the same trace. This post is a reference in three parts. The first defines each metric, shows how to compute it and says when it misleads. The second is a table of the agent benchmarks that matter in November 2026, with the metric each one reports and the state it is in, read from each benchmark's own site or paper on November 13, 2026. The third covers which of those metrics you can compute on your own agent's traces. For the process around them, see how to evaluate AI agents https://www.evalshift.dev/blog/how-to-evaluate-ai-agents . A disclosure: I build EvalShift, an eval tool that computes some of these, and it has a short section near the end. Agent evaluation metrics at a glance | Metric | What it grades | Computed from | Misleads when | |---|---|---|---| | Task success rate, resolved rate | The end state after the run | A verifier: tests, a database comparison, an answer key | The verifier accepts wrong work or rejects right work | | Partial score | How much of the task got done | Checkpoints inside the task | It is read as completion | | pass@k | Whether any of k attempts succeeded | Repeated trials | The user gets one attempt | | pass^k | Whether all k attempts succeeded | Repeated trials | k is not stated | | Tool selection accuracy | Which tools were called | The trace and a reference | The reference is one valid path among several | | Tool argument accuracy | The values passed to each tool | The trace and a reference | Every field is compared as an exact string | | Trajectory match | The sequence of calls | The trace and a reference | The matching mode is not named | | Rule violations | Steps that must never happen, or never be skipped | The trace and written rules | It is reported as a rate | | Steps and tool calls | The work per task | The trace | The step limit differs between runs | | Cost per successful task | Spend per resolved task | Token usage, prices and the outcome | It is reported per run | | Latency | Time to first token and to a finished task | Timestamps | Only the average is shown | | Utility under attack, attack success rate | Behavior when tool results carry injected instructions | An attack suite | The attacks are older than the model | Outcome metrics: did the task get done Task success rate and resolved rate Task success rate is the share of tasks whose end state passes a verifier. Benchmarks give it different names. SWE-bench Pro ↗ https://labs.scale.com/leaderboard/swe bench pro public calls it resolve rate: a patch counts only if the tests that failed before the change now pass and the tests that passed before still pass. Terminal-Bench ↗ https://www.tbench.ai/leaderboard calls it resolution rate. τ-bench calls it reward: 1 when the final database is identical to the expected one and the agent's replies contain the required information, otherwise 0 Yao et al., 2024 ↗ https://arxiv.org/abs/2406.12045 . It is the metric to gate on, and it is only as good as the verifier. Two audits from 2026 show errors in both directions: - +OpenAI audited ↗ https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ the 27.6% of SWE-bench Verified problems that models most often failed, and found that at least 59.4% of them had flawed tests that reject correct solutions. - +Terminal-Bench regraded ↗ https://www.tbench.ai/news/terminal-bench-4-1 11,550 saved trials with fixed verifiers for version 4.1. The score changed on 134 of them, 1.16%: 89 were correct solutions the old verifier had rejected, and 45 were wrong ones it had accepted. A success can also hide a rule break. The τ-bench paper says so about its own reward: an agent that issues a return without asking the user to confirm can still score 1, although it violated the policy. That is why rule violations are a separate metric below. Partial score and checkpoints Binary success is harsh on long tasks, so some benchmarks add partial credit: a list of checkpoints per task and a score for the fraction reached. OSWorld 2.0 ↗ https://osworld-v2.xlang.ai/ defines an average of 27.25 scoring checkpoints per task and reports a partial score next to binary completion. Anthropic's guide to agent evals ↗ https://anthropic.com/engineering/demystifying-evals-for-ai-agents recommends the same for multi-part tasks: a support agent that verifies the customer and finds the problem, then fails the refund, did better than one that failed at once. A partial score shows where a run stopped. It misleads when read as completion: the best agent in the OSWorld 2.0 paper ↗ https://arxiv.org/abs/2606.29537 had a partial score of 54.8% and completed 20.6% of tasks. Report both, and gate on the binary one. pass@k and pass^k One trial per task measures luck as much as behavior. Two metrics describe what happens over k trials, and they point in opposite directions: - + pass@k is the chance that at least one of k trials succeeds. It comes from code generation, where tests can pick the working attempt out of several. - + pass^k is the chance that all k trials succeed. The τ-bench paper introduced it for agents that talk to customers, because each customer gets one trial. The paper gives estimators for both. If a task ran n times and succeeded c times, then, averaged over tasks: pass^k = C c, k / C n, k pass@k = 1 - C n - c, k / C n, k Here C a, b is the number of ways to choose b items from a. Take one task that passed 4 of 5 trials: | k | pass^k | pass@k | |---|---|---| | 1 | 80% | 80% | | 2 | 60% | 100% | | 3 | 40% | 100% | | 4 | 20% | 100% | | 5 | 0% | 100% | The two meet at k = 1, where both are the plain success rate, and separate from there. In the paper, a gpt-4o agent scored 61.2% on retail tasks at pass^1 and under 25% at pass^8. A pass^k figure means nothing without its k, and it needs at least k trials per task, which is why most leaderboards show only pass^1. AI evals for agents https://www.evalshift.dev/blog/ai-evals-for-agents covers when to use which of the two. Trajectory metrics: how the agent got there Trajectory metrics read the trace: the tool calls, in order, with their arguments. They need no environment and no verifier, only a reference to compare against. That makes them the cheapest agent metrics to compute and the easiest to over-trust, because an agent can reach a correct outcome by a path the reference did not anticipate. Tool selection accuracy Tool selection accuracy asks whether the agent called the right tools. There is no single definition, and three metrics from two common libraries disagree by default: | Library | Metric | Default behavior | |---|---|---| | Ragas ↗ https://docs.ragas.io/en/stable/concepts/metrics/available metrics/agents/ | Tool Call Accuracy | Calls must match the reference in order, and a sequence mismatch sets the score to 0. Arguments are checked, with partial credit per call. | | Ragas ↗ https://docs.ragas.io/en/stable/concepts/metrics/available metrics/agents/ | Tool Call F1 | Order is ignored. A call matches only if its name and parameters match exactly. | | DeepEval ↗ https://deepeval.com/docs/metrics-tool-correctness | Tool Correctness | Tool names only, order ignored: correctly used tools divided by the total number of tools called. | Run the three on one trace that has the right calls and arguments in a different order from the reference, and you get 0 from the first and a full score from the other two. When someone quotes "tool-call accuracy 92%", ask which of these it is. Tool argument accuracy Argument accuracy compares the values passed to each matched call with reference values. The choices that change the number are how calls are paired, whether a missing optional field counts as wrong, and how each field is compared: exact for identifiers, a tolerance for amounts, something looser for free text. BFCL, the function-calling benchmark, checks single-turn calls by matching the call's abstract syntax tree ↗ https://gorilla.cs.berkeley.edu/blogs/15 bfcl v4 web search.html , and multi-turn tasks by the state transitions they cause. It misleads when every field gets exact string comparison: a search query worded differently is not a wrong order ID. Evaluating agent tool calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls goes through per-field strategies. Trajectory match Trajectory match compares the whole sequence of calls with a reference sequence. The variants differ in what they forgive: | Variant | Passes when | Vertex AI metric | agentevals mode | |---|---|---|---| | Exact | The sequence is identical to the reference | trajectory exact match | strict | | In order | Every reference call appears, in the same order; extra calls allowed | trajectory in order match | | | Any order | Every reference call appears; extra calls allowed | trajectory any order match | superset | | Same calls | The same calls as the reference, in any order | | unordered | | No extras | Every call the agent made is in the reference | | subset | | Precision | Share of the agent's calls that appear in the reference | trajectory precision | | | Recall | Share of the reference calls the agent made | trajectory recall | | The names come from Google's Vertex AI agent evaluation docs ↗ https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-agents and LangChain's agentevals ↗ https://docs.langchain.com/langsmith/trajectory-evals . The first five are pass or fail per run, and precision and recall are fractions. LangChain also offers an LLM judge that reads the trajectory against a rubric, with or without a reference. Exact match is the strictest and the one to use least: it fails every run that did an extra lookup or reordered two independent calls. Precision and recall work as a pair. Low precision means wasted calls, and low recall means skipped ones. Policy and rule violations Some steps are rules: verify identity before any write, look up the policy before a refund, charge a card at most once. A rule violation metric counts the runs that broke one. It is graded on the trace against written rules and needs no reference path, so one rule applies to every task you will ever add. Report it as a count and budget it at zero. A 98% compliance rate still means two customers in a hundred got a refund with no policy check. This is the metric that catches the τ-bench case above, where the outcome matched and a required step was missing. Efficiency metrics: steps, cost and latency Steps and tool calls Step count is the number of model turns or tool calls a task took. On its own it is a diagnostic: a rising count on unchanged tasks usually means retries or loops. Benchmarks treat the limit as part of the test, and the limits differ widely. OSWorld 2.0 reports completion at a 500-step budget. Most SWE-bench Pro entries ran with a 250-turn limit and no cost cap, where entries on its deprecated board were capped at 50 turns. Terminal-Bench 4.0 set every task to a flat eight-hour agent timeout ↗ https://www.tbench.ai/news/terminal-bench-4-0 . A success rate always carries a step budget, stated or not. Your production loop has one too. Cost per successful task Cost per successful task is everything spent on a set of runs, failed ones included, divided by the number of runs that succeeded. Cost per run leaves out the failures, which is how a cheaper model with a lower success rate ends up costing more per resolved task. How to evaluate AI agents https://www.evalshift.dev/blog/how-to-evaluate-ai-agents has a worked example. Benchmarks have started to print cost next to the score. Terminal-Bench's leaderboard has cost and token columns, BFCL lists an estimated cost for the whole benchmark, and Princeton's Holistic Agent Leaderboard ↗ https://hal.cs.princeton.edu/ reports accuracy with total cost and draws the frontier between them. The argument for doing so is in AI Agents That Matter ↗ https://arxiv.org/abs/2407.01502 , which found that a focus on accuracy alone had produced agents that were needlessly complex and costly, and proposed optimizing the two together. Latency For an agent, latency is at least two numbers: time to first token, which is what a user waiting for a reply feels, and time to a finished task, which grows with every tool call. Anthropic's guide tracks time to first token, output tokens per second and time to last token on each transcript. Report a percentile as well as the mean, because the runs that loop are the ones users remember. Safety metrics Safety metrics measure what an agent does when something in its context is trying to misuse it. The clearest definitions come from AgentDojo ↗ https://arxiv.org/abs/2406.13352 , a benchmark of 97 tasks and 629 security test cases in which tool results contain injected instructions: - + Benign utility: the share of user tasks the agent solves with no attack present. - + Utility under attack: the share of cases where the agent still solves the user's task with an injection present, without adversarial side effects. - + Targeted attack success rate: the share of cases where the attacker's goal is met. All three are computed by deterministic checks on the environment state, not by an LLM judge. The paper's reason is that an injection able to hijack the agent could hijack a judge model too. Report them together, because an agent that refuses every task has an attack success rate of zero and a utility of zero. OSWorld 2.0 takes a different route and attaches a safety report to each task that audits its safety-sensitive steps. AI agent benchmarks in November 2026 and the metric each one reports Every row was read on November 13, 2026 from the benchmark's own site or paper. | Benchmark | What the agent does | Metric reported | State | |---|---|---|---| | SWE-bench Verified ↗ https://www.swebench.com/verified.html | Fixes real GitHub issues, 500 human-validated tasks | Share resolved, graded by the repository's tests | Contaminated. OpenAI stopped reporting it on February 23, 2026 | | SWE-bench Pro ↗ https://labs.scale.com/leaderboard/swe bench pro public | The same job on 1,865 tasks from 41 repositories: 731 public, 276 private, 858 held out | Resolve rate with a confidence interval | Active. Top public entry 61.50 ± 3.10 | | Terminal-Bench 4.1 ↗ https://www.tbench.ai/leaderboard | Completes tasks in a terminal inside a container | Resolution rate with a 95% interval, plus cost and tokens | Maintained, versioned and regraded | | τ-bench ↗ https://taubench.com/ , now τ²-bench and τ³-bench | Serves a simulated customer under a written policy, with tools | pass^1 on the board; text submissions store pass^1 to pass^4 | Maintained. τ³-bench added banking and voice in 2026 | | GAIA ↗ https://huggingface.co/spaces/gaia-benchmark/leaderboard | Answers 466 questions that need browsing, files and tools | Share of answers that match, on a private test set | Saturated. Top entries at 94.35%, human respondents at 92% | | WebArena-Verified ↗ https://github.com/ServiceNow/webarena-verified | Completes 812 tasks on self-hosted websites | Task success, by deterministic evaluators | Re-audited release of WebArena, with a 258-task hard subset | | OSWorld-Verified ↗ https://osworld-v1.xlang.ai/ | Completes 369 desktop tasks | Success rate, by execution-based scripts | Superseded by OSWorld 2.0 in June 2026 | | OSWorld 2.0 ↗ https://osworld-v2.xlang.ai/ | Completes 108 long desktop workflows | Binary completion at 500 steps, plus a partial score | New. Best agent in the paper: 20.6% completion | | BFCL V4 ↗ https://gorilla.cs.berkeley.edu/leaderboard.html | Calls functions: single-turn, multi-turn, web search and memory | Overall accuracy, plus cost and latency | Last updated April 12, 2026 | | AgentDojo ↗ https://agentdojo.spylab.ai/results/ | Does 97 tool-use tasks while tool results carry injections | Utility, utility under attack, targeted attack success rate | Stale. Newest result row is dated February 24, 2025 | | METR time horizons ↗ https://metr.org/time-horizons/ | Does software, ML and security tasks of known human duration | 50% and 80% time horizon | Frozen. The page says it is no longer actively updated | | Holistic Agent Leaderboard ↗ https://hal.cs.princeton.edu/ | Runs nine existing benchmarks in a shared harness | Accuracy and total cost | Paused for new models | Some notes on the rows: - + SWE-bench Verified is the clearest case of a benchmark wearing out. Anthropic's guide describes frontier models as nearing saturation above 80%. On February 23, 2026 OpenAI stopped reporting it, asked other developers to do the same, and recommended SWE-bench Pro until something better exists. - + GAIA released its questions and kept the answers to 300 of them private for the leaderboard. In the paper, human respondents scored 92% and GPT-4 with plugins 15% Mialon et al., 2023 ↗ https://arxiv.org/abs/2311.12983 . The board now says its validation leaderboard is closed because it was no longer informative, and that entries report only each team's best run. - + τ-bench is the only family here built around reliability, and its front page still shows pass^1. The submission guide ↗ https://github.com/sierra-research/tau2-bench/blob/main/docs/leaderboard-submission.md strongly prefers at least four trials per domain for text results, and scores the newer benchmark formats at one trial. τ³-bench shipped with more than 75 task fixes ↗ https://github.com/sierra-research/tau2-bench , and the top pass^1 on its banking domain is 55.2%. - + OSWorld shows how a benchmark gets replaced. Humans complete 72.36% of the original tasks, and a task there takes an agent about 30 tool calls. The 2.0 tasks take a person a median of 1.6 hours and averaged 318 tool calls for Claude Opus 4.7 in the paper. - + METR's time horizon is the only metric here in units of human time: the task length at which a fitted curve predicts the agent succeeds half the time. Two patterns are visible in the table. Almost every row reports an average success rate, and only the τ-bench family has a column for whether the agent succeeds every time. And half of the twelve are contaminated, saturated, superseded, stale, frozen or paused, while three of the rest were re-audited or regraded to stay usable. For what the current boards say about which model is ahead, and five reasons a score doesn't carry over to your agent, see LLM benchmarks https://www.evalshift.dev/blog/llm-benchmarks . Which benchmark metrics you can compute on your own agent Most of them, and the ones that need only traces are the cheapest. | Metric | From your traces alone? | What else it takes | |---|---|---| | Tool selection, arguments, trajectory match | Yes | Reference calls, which recorded production runs already contain | | Rule violations | Yes | The rules, written once | | Steps, tokens, cost, latency | Yes | Token counts and timestamps on each call | | pass@k and pass^k | Yes, if you rerun | k or more trials per task, and a definition of success | | Task success or resolved rate | No | A verifier and an environment it can inspect: tests, a sandbox database, a staging system | | Partial score | No | Checkpoints written per task | | Utility under attack, attack success rate | No | An attack suite built for your tools | | Time horizon | No | Human completion times for every task | | Human baseline, leaderboard rank | No | People, and other teams' agents on your tasks | The trace-only metrics fit in a few lines. This is the whole of pass^k, pass@k, cost per successful task and trajectory precision and recall: python from math import comb from statistics import mean def pass hat k trials, k : """pass^k: the chance that k trials of one task all succeed, averaged over tasks. trials maps a task id to its list of results, one True or False per trial.""" return mean comb sum t , k / comb len t , k for t in trials.values def pass at k trials, k : """pass@k: the chance that at least one of k trials succeeds, averaged over tasks.""" return mean 1 - comb len t - sum t , k / comb len t , k for t in trials.values def cost per success runs : """Everything spent, failed runs included, divided by the runs that succeeded.""" wins = sum run "success" for run in runs return sum run "cost usd" for run in runs / wins if wins else float "inf" def trajectory precision recall called, reference : """Tool names only, order ignored: the loosest trajectory comparison.""" precision = sum name in reference for name in called / len called if called else 0.0 recall = sum name in called for name in reference / len reference if reference else 0.0 return precision, recall Two cautions about the "yes" rows. A reference trajectory taken from a production run is what one model did once, not the only correct path, so use the forgiving match variants. And pass^k needs a success signal per trial. Without a verifier, "success" can only mean the trace met its checks, which is a weaker claim than a resolved task. The "no" rows share one cause: they need something outside the trace. A resolved rate needs the world the agent acted on, and a time horizon needs people who did the same tasks against a clock. The benchmark authors built that environment and verifier, and you would have to build yours. The inputs for the trace metrics already exist if you log production runs. LLM monitoring https://www.evalshift.dev/blog/llm-monitoring covers which signals to record. Agent metrics in EvalShift Disclosure again: I'm the founder. EvalShift https://www.evalshift.dev/docs compares two versions of an agent, usually the current model and a candidate, on the same recorded cases. In terms of this post, it computes the trace-only rows for both sides and tests whether the difference is larger than the noise: - + Tool selection in two forms: conformance to the expected calls, in order with expected or order-insensitive with expected set , and divergence of the candidate from the current model. - + Tool arguments per field, against the current model or, with against: expected , against recorded ground truth. - + Rule violations through trace invariants , with rule types for forbidden tools, required tools, order, call counts and argument schemas. max invariant violations budgets them as a count, and its default is 0. - + Cost and latency per example, budgeted with max cost increase and max latency increase . - + Repeated trials with defaults.samples per example , up to 20 per case. What it does not compute: it replays recorded tool results and does not execute your tools, so there is no resolved rate, no partial score and no attack success rate. Repeated samples are averaged per case, and it does not report pass^k. The evaluators reference https://www.evalshift.dev/docs/evaluators and the agents docs https://www.evalshift.dev/docs/agents list every option, and migration policy https://www.evalshift.dev/docs/migration-policy lists the budgets. FAQ What are agent evaluation metrics? Agent evaluation metrics describe how an AI agent behaved on a task. They fall into four groups: outcome metrics such as task success rate and pass^k, trajectory metrics such as tool selection accuracy, argument accuracy and rule violations, efficiency metrics such as steps, cost per successful task and latency, and safety metrics such as attack success rate. What is the difference between pass@k and pass^k? pass@k is the chance that at least one of k trials of a task succeeds, and pass^k is the chance that all k succeed. For a task that passed 4 of 5 trials, pass@2 is 100% and pass^2 is 60%. Use pass^k for agents that act for a user, who gets one attempt. What are the main AI agent benchmarks? As of November 2026: SWE-bench Verified and SWE-bench Pro for fixing code, Terminal-Bench for terminal tasks, τ-bench for customer service under a policy, GAIA for assistant questions, WebArena for websites, OSWorld for desktop use, BFCL for function calling, AgentDojo for prompt injection, and METR's time horizons for long tasks. Is SWE-bench Verified still a useful benchmark? Less than it was. OpenAI stopped reporting it on February 23, 2026, after finding flawed tests in at least 59.4% of the problems it audited and evidence that models had seen the tasks in training. OpenAI recommends SWE-bench Pro, which keeps private and held-out task sets. How do you measure tool-call accuracy for an AI agent? Compare the tools the agent called with a reference, and state the matching rule, because libraries differ. By default, Ragas Tool Call Accuracy scores 0 when the order differs from the reference, and DeepEval Tool Correctness ignores order and compares tool names only. Report arguments separately from tool names. What is trajectory evaluation? Trajectory evaluation grades the sequence of tool calls an agent made, not only its final answer. Variants range from exact match with a reference sequence to in-order match, any-order match, and precision and recall over the calls. Use the forgiving variants by default, because agents reach correct outcomes by more than one path. Can you compute benchmark metrics on your own agent? Tool selection, argument accuracy, trajectory match, rule violations, steps, cost and latency can be computed from logged traces, and pass^k from repeated runs. A resolved rate needs a verifier and an environment, a time horizon needs human completion times, and attack success rate needs an attack suite. Keep reading - + How to evaluate AI agents https://www.evalshift.dev/blog/how-to-evaluate-ai-agents : the process around these metrics, with levels, environments and a release scorecard. - + LLM benchmarks https://www.evalshift.dev/blog/llm-benchmarks : what the leaderboards say in November 2026, and why a score doesn't predict your agent. - + AI evals for agents https://www.evalshift.dev/blog/ai-evals-for-agents : how to write the tasks and graders that produce these numbers. - + Evaluating agent tool calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls : the trajectory metrics in depth. - + How many eval cases do you need? https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need : how many tasks a metric needs before a difference means something.