{"slug": "agent-evaluation-metrics-and-ai-agent-benchmarks-what-each-one-measures", "title": "Agent evaluation metrics and AI agent benchmarks: what each one measures", "summary": "A reference published in November 2026 defines agent evaluation metrics with formulas and maps them to the AI agent benchmarks that report them, grouping the metrics into outcome, trajectory, efficiency and safety categories. The reference states that identical metric names can diverge across implementations, noting that two popular eval libraries can return 0 and 100% for the same tool-call trace, and that benchmark task success and dashboard task success can differ in grader, number of attempts and step limit. The benchmark table was read from each benchmark's own site or paper on November 13, 2026, and the author discloses building EvalShift, an eval tool that computes some of the metrics.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# Agent evaluation metrics and AI agent benchmarks: what each one measures\n\nAgent evaluation metrics defined with formulas, the AI agent benchmarks that report them as of November 2026, and which ones you can compute yourself.\n\nAgent evaluation metrics are the numbers that describe how an AI agent behaved on a task: whether it finished, which tools it called and with what arguments, whether it broke a rule, how many steps and dollars it used, and how often it does the same thing twice. They fall into four groups: outcome, trajectory, efficiency and safety. An AI agent benchmark is a fixed task set that reports one or two of them for a model, usually an average success rate.\n\n\"Task success\" on a benchmark and \"task success\" on your dashboard can share a name and little else: a different grader, a different number of attempts, a different step limit. The same goes for tool-call accuracy, where two popular eval libraries can return 0 and 100% for the same trace.\n\nThis post is a reference in three parts. The first defines each metric, shows how to compute it\nand says when it misleads. The second is a table of the agent benchmarks that matter in November\n2026, with the metric each one reports and the state it is in, read from each benchmark's own\nsite or paper on November 13, 2026. The third covers which of those metrics you can compute on\nyour own agent's traces. For the process around them, see\n[how to evaluate AI agents](https://www.evalshift.dev/blog/how-to-evaluate-ai-agents). A disclosure: I build EvalShift, an\neval tool that computes some of these, and it has a short section near the end.\n\n## Agent evaluation metrics at a glance\n\n| Metric | What it grades | Computed from | Misleads when | \n|---|---|---|---|\n| Task success rate, resolved rate | The end state after the run | A verifier: tests, a database comparison, an answer key | The verifier accepts wrong work or rejects right work | \n| Partial score | How much of the task got done | Checkpoints inside the task | It is read as completion | \n| pass@k | Whether any of k attempts succeeded | Repeated trials | The user gets one attempt | \n| pass^k | Whether all k attempts succeeded | Repeated trials | k is not stated | \n| Tool selection accuracy | Which tools were called | The trace and a reference | The reference is one valid path among several | \n| Tool argument accuracy | The values passed to each tool | The trace and a reference | Every field is compared as an exact string | \n| Trajectory match | The sequence of calls | The trace and a reference | The matching mode is not named | \n| Rule violations | Steps that must never happen, or never be skipped | The trace and written rules | It is reported as a rate | \n| Steps and tool calls | The work per task | The trace | The step limit differs between runs | \n| Cost per successful task | Spend per resolved task | Token usage, prices and the outcome | It is reported per run | \n| Latency | Time to first token and to a finished task | Timestamps | Only the average is shown | \n| Utility under attack, attack success rate | Behavior when tool results carry injected instructions | An attack suite | The attacks are older than the model | \n\n## Outcome metrics: did the task get done\n\n### Task success rate and resolved rate\n\nTask success rate is the share of tasks whose end state passes a verifier. Benchmarks give it\ndifferent names. [SWE-bench Pro ↗](https://labs.scale.com/leaderboard/swe_bench_pro_public) calls it\nresolve rate: a patch counts only if the tests that failed before the change now pass and the\ntests that passed before still pass. [Terminal-Bench ↗](https://www.tbench.ai/leaderboard) calls it\nresolution rate. τ-bench calls it reward: 1 when the final database is identical to the expected\none and the agent's replies contain the required information, otherwise 0\n([Yao et al., 2024 ↗](https://arxiv.org/abs/2406.12045)).\n\nIt is the metric to gate on, and it is only as good as the verifier. Two audits from 2026 show errors in both directions:\n\n- +OpenAI [audited ↗](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) the\n27.6% of SWE-bench Verified problems that models most often failed, and found that at least\n59.4% of them had flawed tests that reject correct solutions.\n- +Terminal-Bench [regraded ↗](https://www.tbench.ai/news/terminal-bench-4-1) 11,550 saved trials\nwith fixed verifiers for version 4.1. The score changed on 134 of them, 1.16%: 89 were correct\nsolutions the old verifier had rejected, and 45 were wrong ones it had accepted.\n\nA success can also hide a rule break. The τ-bench paper says so about its own reward: an agent that issues a return without asking the user to confirm can still score 1, although it violated the policy. That is why rule violations are a separate metric below.\n\n### Partial score and checkpoints\n\nBinary success is harsh on long tasks, so some benchmarks add partial credit: a list of\ncheckpoints per task and a score for the fraction reached.\n[OSWorld 2.0 ↗](https://osworld-v2.xlang.ai/) defines an average of 27.25 scoring checkpoints per\ntask and reports a partial score next to binary completion. Anthropic's\n[guide to agent evals ↗](https://anthropic.com/engineering/demystifying-evals-for-ai-agents)\nrecommends the same for multi-part tasks: a support agent that verifies the customer and finds\nthe problem, then fails the refund, did better than one that failed at once.\n\nA partial score shows where a run stopped. It misleads when read as completion: the best agent\nin the [OSWorld 2.0 paper ↗](https://arxiv.org/abs/2606.29537) had a partial score of 54.8% and\ncompleted 20.6% of tasks. Report both, and gate on the binary one.\n\n### pass@k and pass^k\n\nOne trial per task measures luck as much as behavior. Two metrics describe what happens over k trials, and they point in opposite directions:\n\n- +**pass@k** is the chance that at least one of k trials succeeds. It comes from code generation,\nwhere tests can pick the working attempt out of several.\n- +**pass^k** is the chance that all k trials succeed. The τ-bench paper introduced it for agents\nthat talk to customers, because each customer gets one trial.\n\nThe paper gives estimators for both. If a task ran *n* times and succeeded *c* times, then,\naveraged over tasks:\n\n```\npass^k = C(c, k) / C(n, k)\npass@k = 1 - C(n - c, k) / C(n, k)\n```\n\nHere C(a, b) is the number of ways to choose b items from a. Take one task that passed 4 of 5 trials:\n\n| k | pass^k | pass@k | \n|---|---|---|\n| 1 | 80% | 80% | \n| 2 | 60% | 100% | \n| 3 | 40% | 100% | \n| 4 | 20% | 100% | \n| 5 | 0% | 100% | \n\nThe two meet at k = 1, where both are the plain success rate, and separate from there. In the\npaper, a `gpt-4o` agent scored 61.2% on retail tasks at pass^1 and under 25% at pass^8.\n\nA pass^k figure means nothing without its k, and it needs at least k trials per task, which is\nwhy most leaderboards show only pass^1. [AI evals for agents](https://www.evalshift.dev/blog/ai-evals-for-agents) covers\nwhen to use which of the two.\n\n## Trajectory metrics: how the agent got there\n\nTrajectory metrics read the trace: the tool calls, in order, with their arguments. They need no environment and no verifier, only a reference to compare against. That makes them the cheapest agent metrics to compute and the easiest to over-trust, because an agent can reach a correct outcome by a path the reference did not anticipate.\n\n### Tool selection accuracy\n\nTool selection accuracy asks whether the agent called the right tools. There is no single definition, and three metrics from two common libraries disagree by default:\n\n| Library | Metric | Default behavior | \n|---|---|---|\n| [Ragas ↗](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/) | Tool Call Accuracy | Calls must match the reference in order, and a sequence mismatch sets the score to 0. Arguments are checked, with partial credit per call. | \n| [Ragas ↗](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/) | Tool Call F1 | Order is ignored. A call matches only if its name and parameters match exactly. | \n| [DeepEval ↗](https://deepeval.com/docs/metrics-tool-correctness) | Tool Correctness | Tool names only, order ignored: correctly used tools divided by the total number of tools called. | \n\nRun the three on one trace that has the right calls and arguments in a different order from the reference, and you get 0 from the first and a full score from the other two. When someone quotes \"tool-call accuracy 92%\", ask which of these it is.\n\n### Tool argument accuracy\n\nArgument accuracy compares the values passed to each matched call with reference values. The\nchoices that change the number are how calls are paired, whether a missing optional field counts\nas wrong, and how each field is compared: exact for identifiers, a tolerance for amounts,\nsomething looser for free text. BFCL, the function-calling benchmark, checks single-turn calls by\n[matching the call's abstract syntax tree ↗](https://gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html),\nand multi-turn tasks by the state transitions they cause.\n\nIt misleads when every field gets exact string comparison: a search query worded differently\nis not a wrong order ID.\n[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) goes through per-field\nstrategies.\n\n### Trajectory match\n\nTrajectory match compares the whole sequence of calls with a reference sequence. The variants differ in what they forgive:\n\n| Variant | Passes when | Vertex AI metric | agentevals mode | \n|---|---|---|---|\n| Exact | The sequence is identical to the reference | `trajectory_exact_match` | `strict` | \n| In order | Every reference call appears, in the same order; extra calls allowed | `trajectory_in_order_match` |  | \n| Any order | Every reference call appears; extra calls allowed | `trajectory_any_order_match` | `superset` | \n| Same calls | The same calls as the reference, in any order |  | `unordered` | \n| No extras | Every call the agent made is in the reference |  | `subset` | \n| Precision | Share of the agent's calls that appear in the reference | `trajectory_precision` |  | \n| Recall | Share of the reference calls the agent made | `trajectory_recall` |  | \n\nThe names come from Google's\n[Vertex AI agent evaluation docs ↗](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-agents)\nand LangChain's [agentevals ↗](https://docs.langchain.com/langsmith/trajectory-evals). The first\nfive are pass or fail per run, and precision and recall are fractions. LangChain also offers an\nLLM judge that reads the trajectory against a rubric, with or without a reference.\n\nExact match is the strictest and the one to use least: it fails every run that did an extra lookup or reordered two independent calls. Precision and recall work as a pair. Low precision means wasted calls, and low recall means skipped ones.\n\n### Policy and rule violations\n\nSome steps are rules: verify identity before any write, look up the policy before a refund, charge a card at most once. A rule violation metric counts the runs that broke one. It is graded on the trace against written rules and needs no reference path, so one rule applies to every task you will ever add.\n\nReport it as a count and budget it at zero. A 98% compliance rate still means two customers in a hundred got a refund with no policy check. This is the metric that catches the τ-bench case above, where the outcome matched and a required step was missing.\n\n## Efficiency metrics: steps, cost and latency\n\n### Steps and tool calls\n\nStep count is the number of model turns or tool calls a task took. On its own it is a\ndiagnostic: a rising count on unchanged tasks usually means retries or loops. Benchmarks treat\nthe limit as part of the test, and the limits differ widely. OSWorld 2.0 reports completion at a\n500-step budget. Most SWE-bench Pro entries ran with a 250-turn limit and no cost cap, where\nentries on its deprecated board were capped at 50 turns. Terminal-Bench 4.0 set every task to a\n[flat eight-hour agent timeout ↗](https://www.tbench.ai/news/terminal-bench-4-0).\n\nA success rate always carries a step budget, stated or not. Your production loop has one too.\n\n### Cost per successful task\n\nCost per successful task is everything spent on a set of runs, failed ones included, divided by\nthe number of runs that succeeded. Cost per run leaves out the failures, which is how a cheaper\nmodel with a lower success rate ends up costing more per resolved task.\n[How to evaluate AI agents](https://www.evalshift.dev/blog/how-to-evaluate-ai-agents) has a worked example.\n\nBenchmarks have started to print cost next to the score. Terminal-Bench's leaderboard has cost\nand token columns, BFCL lists an estimated cost for the whole benchmark, and Princeton's\n[Holistic Agent Leaderboard ↗](https://hal.cs.princeton.edu/) reports accuracy with total cost and\ndraws the frontier between them. The argument for doing so is in\n[AI Agents That Matter ↗](https://arxiv.org/abs/2407.01502), which found that a focus on accuracy\nalone had produced agents that were needlessly complex and costly, and proposed optimizing the\ntwo together.\n\n### Latency\n\nFor an agent, latency is at least two numbers: time to first token, which is what a user waiting for a reply feels, and time to a finished task, which grows with every tool call. Anthropic's guide tracks time to first token, output tokens per second and time to last token on each transcript. Report a percentile as well as the mean, because the runs that loop are the ones users remember.\n\n## Safety metrics\n\nSafety metrics measure what an agent does when something in its context is trying to misuse it.\nThe clearest definitions come from [AgentDojo ↗](https://arxiv.org/abs/2406.13352), a benchmark of\n97 tasks and 629 security test cases in which tool results contain injected instructions:\n\n- +**Benign utility:** the share of user tasks the agent solves with no attack present.\n- +**Utility under attack:** the share of cases where the agent still solves the user's task with\nan injection present, without adversarial side effects.\n- +**Targeted attack success rate:** the share of cases where the attacker's goal is met.\n\nAll three are computed by deterministic checks on the environment state, not by an LLM judge. The paper's reason is that an injection able to hijack the agent could hijack a judge model too.\n\nReport them together, because an agent that refuses every task has an attack success rate of zero and a utility of zero. OSWorld 2.0 takes a different route and attaches a safety report to each task that audits its safety-sensitive steps.\n\n## AI agent benchmarks in November 2026 and the metric each one reports\n\nEvery row was read on November 13, 2026 from the benchmark's own site or paper.\n\n| Benchmark | What the agent does | Metric reported | State | \n|---|---|---|---|\n| [SWE-bench Verified ↗](https://www.swebench.com/verified.html) | Fixes real GitHub issues, 500 human-validated tasks | Share resolved, graded by the repository's tests | Contaminated. OpenAI stopped reporting it on February 23, 2026 | \n| [SWE-bench Pro ↗](https://labs.scale.com/leaderboard/swe_bench_pro_public) | The same job on 1,865 tasks from 41 repositories: 731 public, 276 private, 858 held out | Resolve rate with a confidence interval | Active. Top public entry 61.50 ± 3.10 | \n| [Terminal-Bench 4.1 ↗](https://www.tbench.ai/leaderboard) | Completes tasks in a terminal inside a container | Resolution rate with a 95% interval, plus cost and tokens | Maintained, versioned and regraded | \n| [τ-bench ↗](https://taubench.com/) , now τ²-bench and τ³-bench | Serves a simulated customer under a written policy, with tools | pass^1 on the board; text submissions store pass^1 to pass^4 | Maintained. τ³-bench added banking and voice in 2026 | \n| [GAIA ↗](https://huggingface.co/spaces/gaia-benchmark/leaderboard) | Answers 466 questions that need browsing, files and tools | Share of answers that match, on a private test set | Saturated. Top entries at 94.35%, human respondents at 92% | \n| [WebArena-Verified ↗](https://github.com/ServiceNow/webarena-verified) | Completes 812 tasks on self-hosted websites | Task success, by deterministic evaluators | Re-audited release of WebArena, with a 258-task hard subset | \n| [OSWorld-Verified ↗](https://osworld-v1.xlang.ai/) | Completes 369 desktop tasks | Success rate, by execution-based scripts | Superseded by OSWorld 2.0 in June 2026 | \n| [OSWorld 2.0 ↗](https://osworld-v2.xlang.ai/) | Completes 108 long desktop workflows | Binary completion at 500 steps, plus a partial score | New. Best agent in the paper: 20.6% completion | \n| [BFCL V4 ↗](https://gorilla.cs.berkeley.edu/leaderboard.html) | Calls functions: single-turn, multi-turn, web search and memory | Overall accuracy, plus cost and latency | Last updated April 12, 2026 | \n| [AgentDojo ↗](https://agentdojo.spylab.ai/results/) | Does 97 tool-use tasks while tool results carry injections | Utility, utility under attack, targeted attack success rate | Stale. Newest result row is dated February 24, 2025 | \n| [METR time horizons ↗](https://metr.org/time-horizons/) | Does software, ML and security tasks of known human duration | 50% and 80% time horizon | Frozen. The page says it is no longer actively updated | \n| [Holistic Agent Leaderboard ↗](https://hal.cs.princeton.edu/) | Runs nine existing benchmarks in a shared harness | Accuracy and total cost | Paused for new models | \n\nSome notes on the rows:\n\n- +**SWE-bench Verified** is the clearest case of a benchmark wearing out. Anthropic's guide\ndescribes frontier models as nearing saturation above 80%. On February 23, 2026 OpenAI\nstopped reporting it, asked other developers to do the same, and recommended SWE-bench Pro\nuntil something better exists.\n- +**GAIA** released its questions and kept the answers to 300 of them private for the\nleaderboard. In the paper, human respondents scored 92% and GPT-4 with plugins 15%\n([Mialon et al., 2023 ↗](https://arxiv.org/abs/2311.12983) ). The board now says its validation\nleaderboard is closed because it was no longer informative, and that entries report only each\nteam's best run.\n- +**τ-bench** is the only family here built around reliability, and its front page still shows\npass^1. The[submission guide ↗](https://github.com/sierra-research/tau2-bench/blob/main/docs/leaderboard-submission.md) strongly prefers at least four trials per domain for text results, and scores the newer\nbenchmark formats at one trial. τ³-bench shipped with more than 75[task fixes ↗](https://github.com/sierra-research/tau2-bench) , and the top pass^1 on its banking\ndomain is 55.2%.\n- +**OSWorld** shows how a benchmark gets replaced. Humans complete 72.36% of the original tasks,\nand a task there takes an agent about 30 tool calls. The 2.0 tasks take a person a median of\n1.6 hours and averaged 318 tool calls for Claude Opus 4.7 in the paper.\n- +**METR's** time horizon is the only metric here in units of human time: the task length at\nwhich a fitted curve predicts the agent succeeds half the time.\n\nTwo patterns are visible in the table. Almost every row reports an average success rate, and\nonly the τ-bench family has a column for whether the agent succeeds every time. And half of the\ntwelve are contaminated, saturated, superseded, stale, frozen or paused, while three of the rest\nwere re-audited or regraded to stay usable. For what the current boards say about which model is\nahead, and five reasons a score doesn't carry over to your agent, see\n[LLM benchmarks](https://www.evalshift.dev/blog/llm-benchmarks).\n\n## Which benchmark metrics you can compute on your own agent\n\nMost of them, and the ones that need only traces are the cheapest.\n\n| Metric | From your traces alone? | What else it takes | \n|---|---|---|\n| Tool selection, arguments, trajectory match | Yes | Reference calls, which recorded production runs already contain | \n| Rule violations | Yes | The rules, written once | \n| Steps, tokens, cost, latency | Yes | Token counts and timestamps on each call | \n| pass@k and pass^k | Yes, if you rerun | k or more trials per task, and a definition of success | \n| Task success or resolved rate | No | A verifier and an environment it can inspect: tests, a sandbox database, a staging system | \n| Partial score | No | Checkpoints written per task | \n| Utility under attack, attack success rate | No | An attack suite built for your tools | \n| Time horizon | No | Human completion times for every task | \n| Human baseline, leaderboard rank | No | People, and other teams' agents on your tasks | \n\nThe trace-only metrics fit in a few lines. This is the whole of pass^k, pass@k, cost per successful task and trajectory precision and recall:\n\n``` python\nfrom math import comb\nfrom statistics import mean\n\ndef pass_hat_k(trials, k):\n    \"\"\"pass^k: the chance that k trials of one task all succeed, averaged over tasks.\n    `trials` maps a task id to its list of results, one True or False per trial.\"\"\"\n    return mean(comb(sum(t), k) / comb(len(t), k) for t in trials.values())\n\ndef pass_at_k(trials, k):\n    \"\"\"pass@k: the chance that at least one of k trials succeeds, averaged over tasks.\"\"\"\n    return mean(1 - comb(len(t) - sum(t), k) / comb(len(t), k) for t in trials.values())\n\ndef cost_per_success(runs):\n    \"\"\"Everything spent, failed runs included, divided by the runs that succeeded.\"\"\"\n    wins = sum(run[\"success\"] for run in runs)\n    return sum(run[\"cost_usd\"] for run in runs) / wins if wins else float(\"inf\")\n\ndef trajectory_precision_recall(called, reference):\n    \"\"\"Tool names only, order ignored: the loosest trajectory comparison.\"\"\"\n    precision = sum(name in reference for name in called) / len(called) if called else 0.0\n    recall = sum(name in called for name in reference) / len(reference) if reference else 0.0\n    return precision, recall\n```\n\nTwo cautions about the \"yes\" rows. A reference trajectory taken from a production run is what one model did once, not the only correct path, so use the forgiving match variants. And pass^k needs a success signal per trial. Without a verifier, \"success\" can only mean the trace met its checks, which is a weaker claim than a resolved task.\n\nThe \"no\" rows share one cause: they need something outside the trace. A resolved rate needs the world the agent acted on, and a time horizon needs people who did the same tasks against a clock. The benchmark authors built that environment and verifier, and you would have to build yours.\n\nThe inputs for the trace metrics already exist if you log production runs.\n[LLM monitoring](https://www.evalshift.dev/blog/llm-monitoring) covers which signals to record.\n\n## Agent metrics in EvalShift\n\nDisclosure again: I'm the founder. [EvalShift](https://www.evalshift.dev/docs) compares two versions of an agent, usually\nthe current model and a candidate, on the same recorded cases. In terms of this post, it computes\nthe trace-only rows for both sides and tests whether the difference is larger than the noise:\n\n- +**Tool selection** in two forms: conformance to the expected calls, in order with`expected` or order-insensitive with`expected_set` , and divergence of the candidate from the current\nmodel.\n- +**Tool arguments** per field, against the current model or, with`against: expected` , against\nrecorded ground truth.\n- +**Rule violations** through`trace_invariants` , with rule types for forbidden tools, required\ntools, order, call counts and argument schemas.`max_invariant_violations` budgets them as a\ncount, and its default is 0.\n- +**Cost and latency** per example, budgeted with`max_cost_increase` and`max_latency_increase` .\n- +**Repeated trials** with`defaults.samples_per_example` , up to 20 per case.\n\nWhat it does not compute: it replays recorded tool results and does not execute your tools, so\nthere is no resolved rate, no partial score and no attack success rate. Repeated samples are\naveraged per case, and it does not report pass^k. The [evaluators reference](https://www.evalshift.dev/docs/evaluators)\nand the [agents docs](https://www.evalshift.dev/docs/agents) list every option, and\n[migration policy](https://www.evalshift.dev/docs/migration-policy) lists the budgets.\n\n## FAQ\n\n**What are agent evaluation metrics?** Agent evaluation metrics describe how an AI agent behaved\non a task. They fall into four groups: outcome metrics such as task success rate and pass^k,\ntrajectory metrics such as tool selection accuracy, argument accuracy and rule violations,\nefficiency metrics such as steps, cost per successful task and latency, and safety metrics such\nas attack success rate.\n\n**What is the difference between pass@k and pass^k?** pass@k is the chance that at least one of k\ntrials of a task succeeds, and pass^k is the chance that all k succeed. For a task that passed 4\nof 5 trials, pass@2 is 100% and pass^2 is 60%. Use pass^k for agents that act for a user, who\ngets one attempt.\n\n**What are the main AI agent benchmarks?** As of November 2026: SWE-bench Verified and SWE-bench\nPro for fixing code, Terminal-Bench for terminal tasks, τ-bench for customer service under a\npolicy, GAIA for assistant questions, WebArena for websites, OSWorld for desktop use, BFCL for\nfunction calling, AgentDojo for prompt injection, and METR's time horizons for long tasks.\n\n**Is SWE-bench Verified still a useful benchmark?** Less than it was. OpenAI stopped reporting it\non February 23, 2026, after finding flawed tests in at least 59.4% of the problems it audited and\nevidence that models had seen the tasks in training. OpenAI recommends SWE-bench Pro, which keeps\nprivate and held-out task sets.\n\n**How do you measure tool-call accuracy for an AI agent?** Compare the tools the agent called\nwith a reference, and state the matching rule, because libraries differ. By default, Ragas Tool\nCall Accuracy scores 0 when the order differs from the reference, and DeepEval Tool Correctness\nignores order and compares tool names only. Report arguments separately from tool names.\n\n**What is trajectory evaluation?** Trajectory evaluation grades the sequence of tool calls an\nagent made, not only its final answer. Variants range from exact match with a reference sequence\nto in-order match, any-order match, and precision and recall over the calls. Use the forgiving\nvariants by default, because agents reach correct outcomes by more than one path.\n\n**Can you compute benchmark metrics on your own agent?** Tool selection, argument accuracy,\ntrajectory match, rule violations, steps, cost and latency can be computed from logged traces,\nand pass^k from repeated runs. A resolved rate needs a verifier and an environment, a time\nhorizon needs human completion times, and attack success rate needs an attack suite.\n\n## Keep reading\n\n- +[How to evaluate AI agents](https://www.evalshift.dev/blog/how-to-evaluate-ai-agents) : the process around these\nmetrics, with levels, environments and a release scorecard.\n- +[LLM benchmarks](https://www.evalshift.dev/blog/llm-benchmarks) : what the leaderboards say in November 2026, and why a\nscore doesn't predict your agent.\n- +[AI evals for agents](https://www.evalshift.dev/blog/ai-evals-for-agents) : how to write the tasks and graders that\nproduce these numbers.\n- +[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) : the trajectory metrics in\ndepth.\n- +[How many eval cases do you need?](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) : how many tasks a\nmetric needs before a difference means something.", "url": "https://wpnews.pro/news/agent-evaluation-metrics-and-ai-agent-benchmarks-what-each-one-measures", "canonical_source": "https://www.evalshift.dev/blog/agent-evaluation-metrics", "published_at": "2026-11-13 21:57:30.246676+00:00", "updated_at": "2026-11-13 21:57:32.559323+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-research", "mlops"], "entities": ["EvalShift", "SWE-bench Pro", "Terminal-Bench", "τ-bench", "Scale AI", "Yao et al."], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/agent-evaluation-metrics-and-ai-agent-benchmarks-what-each-one-measures", "markdown": "https://wpnews.pro/news/agent-evaluation-metrics-and-ai-agent-benchmarks-what-each-one-measures.md", "text": "https://wpnews.pro/news/agent-evaluation-metrics-and-ai-agent-benchmarks-what-each-one-measures.txt", "jsonld": "https://wpnews.pro/news/agent-evaluation-metrics-and-ai-agent-benchmarks-what-each-one-measures.jsonld"}}