{"slug": "ai-evals-for-agents-what-they-are-and-how-to-write-them", "title": "AI evals for agents: what they are and how to write them", "summary": "A guide published on Anthropic's engineering blog in January 2026, \"Demystifying evals for AI agents,\" sets out a shared vocabulary for agent evaluations and argues that agent evals must grade the full run — tool calls, arguments, resulting state and final reply — rather than a single response, because an agent can reply \"your refund is on its way\" without ever calling the refund tool and a text-only eval will mark that run as a pass. The guide defines six components (task, trial, grader, transcript, outcome, suite), distinguishes LLM evals from agent evals, and describes three grader types — code, model (LLM judge) and human — with model judges costing about $1 per 1,000 calls on a small judge. It then walks through writing an agent eval in seven steps, using a 120-task support suite and a worked refund example (order A1042, a $59.00 refund) in which a single task carries several graders so each failure mode is distinguishable.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# AI evals for agents: what they are and how to write them\n\nAI evals are tests that grade what a model or agent did. How agent evals differ from LLM evals, the three grader types, and how to write one in 7 steps.\n\nAn AI eval is a test for an AI system: you give it an input, let it run, and grade what it did with code, with another model, or with a person. LLM evals grade a single response. Evals for AI agents grade a whole run: the tools the agent called, the arguments it passed, the state it left behind and the reply it ended with. That difference decides how you write them, because an agent can reply \"your refund is on its way\" without ever calling the refund tool, and an eval that reads only the reply will mark that run as a pass.\n\nThis guide defines the parts of an eval, explains how agent evals differ from LLM evals, covers the three kinds of grader, and then walks through writing an eval for an agent in seven steps, with a worked example you can adapt.\n\n## What an AI eval is made of\n\nAnthropic's guide to agent evals\n([Demystifying evals for AI agents ↗](https://anthropic.com/engineering/demystifying-evals-for-ai-agents),\nJanuary 2026) uses a vocabulary most eval tools now share. It is worth adopting, because \"the eval\nfailed\" can mean any of these:\n\n| Term | What it is | Example for a support agent | \n|---|---|---|\n| Task | One test case: an input plus what success means | \"Order A1042 arrived broken, can I get my money back?\" Success: one refund of $59.00 on A1042 | \n| Trial | One attempt at a task. Agents vary between runs, so a task gets several | Run 3 of 5 | \n| Grader | Logic that scores one aspect of a trial | \"Exactly one refund exists for A1042\" | \n| Transcript (trace) | The full record of a trial: messages, tool calls, tool results, the final reply | `check_refund_policy` →`issue_refund(order_id=\"A1042\", amount_usd=59.0)` → reply | \n| Outcome | The state of the world after the trial | The row in the refunds table, not the sentence claiming it | \n| Suite | The set of tasks you run together | 120 support tasks, tagged by route | \n\nA task can have several graders, and usually should. One that checks the refund exists, one that checks policy was looked up first, and one that checks the reply explains the timeline will each fail for a different reason, and you want to know which.\n\n## LLM evals vs agent evals\n\nThe terms overlap and are often used interchangeably, but they grade different things.\n\n|  | LLM eval | Agent eval | \n|---|---|---|\n| Unit graded | One response to one prompt | A run: several model calls, tool calls and results | \n| What can go wrong | Wrong, off-topic or malformed text | Wrong tool, wrong arguments, skipped step, repeated action, false claim of success, plus everything an LLM eval catches | \n| Where the evidence is | The output text | The trace and the outcome, then the text | \n| Determinism | Varies between samples | Varies more: one different tool call changes every step after it | \n\nThe practical consequence: an agent's worst regressions read as correct. A model that stops\ncalling `notify_security_team`, or reports an address change it never made, still writes fluent,\non-topic prose. Every eval that scores the text alone passes it.\n[What actually breaks when you switch LLMs](https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms) goes through\nthe full change surface.\n\n## The three kinds of grader\n\nEvery eval grader is one of three kinds, and a good suite uses all three for different jobs.\n\n| Grader | Examples | Cost | Strength | Weakness | \n|---|---|---|---|---|\n| Code | Schema validation, exact match, regex, tool-call checks, database state | Free, instant | Deterministic, cannot be argued with | Only checks what you can specify exactly | \n| Model (LLM judge) | Rubric scoring, pairwise comparison, natural-language assertions | About $1 per 1,000 calls on a small judge | Handles open-ended text | Biased toward position, length and its own phrasing; needs calibration | \n| Human | Expert review, spot checks, labeling | Reviewer hours | The ground truth everything else is calibrated against | Slow, expensive, drifts with fatigue | \n\nThe order matters. Use code for everything code can check, a model for the quality of prose that\nis left, and people to write the criteria, label a calibration sample and read the failures.\n[LLM-as-a-judge](https://www.evalshift.dev/blog/llm-as-a-judge) covers the model grader in depth, including how accurate\njudges are and how to validate one.\n\n## Capability evals vs regression evals\n\nEvals answer one of two questions, and mixing them up makes the numbers hard to read.\n\n- +**Capability evals** ask what the agent can do. They should start with a low pass rate, because\na suite the agent already passes cannot show progress. You use them while building.\n- +**Regression evals** ask whether the agent still does what it did yesterday. They should sit near\n100%, so any drop is a signal. You run them on every change to the model, the prompt or the\ntools.\n\nA capability task the agent passes reliably graduates into the regression suite. Most teams building on a model provider spend more of their eval effort on regression than they expect, because the things that change most often are not their code: a model upgrade, a provider switch, a deprecated model with a retirement date.\n\n## Agents are not deterministic: pass@k and pass^k\n\nRun the same agent on the same task five times and you can get five different traces. One trial per task measures luck as much as behavior. Two metrics describe the spread:\n\n- +**pass@k:** the chance that at least one of k trials succeeds. It rises with k, and it is the\nright number when one success is enough, as with a coding agent whose output a person will\nreview.\n- +**pass^k:** the chance that all k trials succeed. It falls with k, and it is the right number for\nan agent customers talk to, because each of them gets one trial.\n\nThe gap is large. In τ-bench, a benchmark of agents serving simulated retail and airline customers\nunder written policies, GPT-4o with function calling solved about 61% of retail tasks on a single\ntrial, and its pass^8, solving the same task in all eight trials, fell to about 25%\n([Yao et al., 2024 ↗](https://arxiv.org/abs/2406.12045)). An agent that succeeds 75% of the time per\ntrial passes three trials in a row only 42% of the time (0.75³).\n\n## How to write an eval for an AI agent\n\nThe seven steps below take you from nothing to a suite you can gate a release on. They assume an agent in or near production. If you are still prototyping, start with step 1 and stop after step 4.\n\n## Step 1: start from real failures and real traffic\n\nDo not start by imagining test cases. Start with the 20 to 50 cases that already went wrong: bug reports, support escalations, transcripts someone flagged. Anthropic's guide makes the same recommendation, because early on, the changes you make have large effects and a small suite detects them.\n\nThen add real traffic. Hand-written cases cover the situations you already thought of; production\nlogs contain the ones you did not. Sample 100 to 300 inputs, stratified by route or intent so a\nrare but expensive route is not drowned out, strip personal data, and drop near-duplicates.\n[Build a golden eval suite from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic)\ncovers capturing those runs, and [how many eval cases you need](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need)\ncovers sizing.\n\nTag every case with its route and any known edge case. Pass rates per tag are what you will act on later.\n\n## Step 2: define success as an outcome, before you write a grader\n\nWrite down what done looks like for each task in terms of the world, not the reply:\n\n- +\"One refund of $59.00 exists on order A1042\", not \"the agent says the refund was issued\".\n- +\"The booking exists for the 14:00 slot and no other booking was created\", not \"the reply confirms the booking\".\n- +\"No tool was called\" for a question the agent should answer from its instructions.\n\nIf you cannot write success down, a grader cannot check it, and neither can a colleague. That is a sign the task is ambiguous, not that you need a cleverer grader. For tasks with a fixed answer, store the expected outcome on the task. For tasks with many valid answers, store the properties every valid answer has.\n\n## Step 3: grade the outcome, and the path only where it is a rule\n\nAgents often reach a correct outcome by a route you did not anticipate: a different lookup order, an extra search, a clarifying question. An eval that requires an exact sequence of tool calls fails those runs and teaches you to ignore failures. Anthropic's advice is to \"grade what the agent produced, not the path it took.\"\n\nThere is one exception, and it matters most for agents that act. Some paths are rules, and breaking them is a failure even when the outcome looks right:\n\n- +Authenticate or verify before any write.\n- +Look up the refund policy before issuing a refund.\n- +Charge a card at most once.\n- +Never call a deprecated or dangerous endpoint.\n- +Keep an amount inside its allowed bounds.\n\nWrite those as checks on the trace, one rule each. Leave everything else free.\n\n## Step 4: use code first, a judge for the rest, people to calibrate\n\nGo through the success definition from step 2 and assign each property to the cheapest grader that can check it.\n\n**Code graders** take the outcome and the trace rules: the refund row exists, its amount is right,\n`check_refund_policy` ran before `issue_refund`, `issue_refund` ran once, the structured output\nparses. A sketch in Python, independent of any framework:\n\n``` python\ndef grade(trial, db):\n    tools = [call.name for call in trial.tool_calls]\n    refunds = db.refunds_for(order_id=\"A1042\")\n\n    refund_issued = len(refunds) == 1 and refunds[0].amount_usd == 59.00\n    policy_first = (\n        \"check_refund_policy\" in tools\n        and \"issue_refund\" in tools\n        and tools.index(\"check_refund_policy\") < tools.index(\"issue_refund\")\n    )\n    # The reply may only claim a refund when one exists.\n    honest_reply = refund_issued or \"refund\" not in trial.final_text.lower()\n\n    return {\n        \"refund_issued\": refund_issued,\n        \"policy_first\": policy_first,\n        \"honest_reply\": honest_reply,\n    }\n```\n\nReturn one result per check, not one combined score. A task that fails `policy_first` and passes\n`refund_issued` is a different bug from one that fails both.\n\n**A model grader** takes what code cannot: whether the reply explains when the money arrives,\nwhether the tone fits an upset customer. Give it one criterion per judge, phrased as observable\nbehavior, with a clear pass condition:\n\n```\nCriterion: Does the reply tell the customer when the refund will reach them?\nIt passes if it gives a timeframe or a date. It fails if it says only that\nthe refund was issued, or gives no timing at all.\nAnswer PASS or FAIL, then one sentence of reasoning.\n```\n\n**People** label 30 to 50 trials by hand and compare their labels with the judge's. Every\ndisagreement is either a vague criterion or a judge you should not trust yet. Keep a new judge\nadvisory, reported but not blocking, until that comparison holds up.\n\n## Step 5: run every task several times\n\nRun each task at least three times, five where the stakes are high, and report pass^k for customer-facing behavior: the share of tasks the agent got right every time. A task that passes two runs out of three is not a pass. It is a flaky behavior your customers will meet one time in three.\n\nSeveral trials also separate noise from regressions. If the current model already disagrees with itself on a task, a different result from a new model on that task proves nothing on its own.\n\n## Step 6: read the transcripts, not just the score\n\nAn aggregate pass rate tells you that something moved, not what. Break results down per tag and per grader, and read every failed transcript in the routes you care about. Most eval bugs show up there first: a grader checking the wrong field, a task with an impossible success condition, a test case that leaked into the prompt.\n\nRead some passes too. A grader that never fails is either checking something trivial or broken.\n\n## Step 7: freeze the suite and run it on every change\n\nA regression suite only proves something if it holds still. Freeze the tasks and graders, version them, and run the suite on every change that can alter behavior: the model name, the prompt, the tool definitions, the agent's code. Comparing two runs on a suite that changed in between compares two moving things.\n\nWhen a change is intended, such as a new route or a corrected expectation, update the suite in its\nown commit, so the next comparison starts from a known baseline. Then wire the run into CI, so a\npull request that regresses the agent fails before it merges.\n[LLM regression testing in CI](https://www.evalshift.dev/blog/llm-regression-testing-in-ci) shows a gate on GitHub Actions.\n\n## Common mistakes in agent evals\n\n- +**Grading the claim instead of the outcome.** \"The agent said it booked the slot\" is not a\nbooking. Check the state, or at least the tool call that would have changed it.\n- +**Requiring an exact path.** Pinning every tool call in order fails valid runs. Pin only the rules\nfrom step 3.\n- +**One averaged score.** \"Quality dropped 3%\" cannot be fixed. \"Twelve tasks on the refund route\nskipped`check_refund_policy` \" can.\n- +**One trial per task.** You will chase noise and miss flakiness.\n- +**A judge nobody validated.** A judge returns a confident number whether or not its criterion\nmakes sense. Calibrate it against human labels first.\n- +**Changing two things at once.** A new model and a rewritten prompt in one comparison leave\nnobody able to say which one moved the score. Change one, measure, then change the other.\n\n## Writing agent evals with EvalShift\n\nEvalShift is built for the regression side of agent evals, and for the change most teams test least: switching or upgrading the model. It replays a frozen suite against the current model and a candidate, grades both sides of every task, and turns the differences into a verdict.\n\n- +**Tasks from real traffic.** The capture SDK records production runs, and`evalshift capture sync` turns them into a golden suite where every task carries its recorded tools and expected tool\ncalls.\n- +**Trace graders, no code required.**`tool_selection` and`tool_arguments` check which tools were\ncalled and with which values, and`trace_invariants` holds both models to the rules from step 3:\n\n```\nevaluators:\n  trace_invariants:\n    - name: refunds_contract\n      owner: \"@support-team\"\n      rules:\n        - id: policy-before-refund\n          type: order\n          before: check_refund_policy\n          after: issue_refund\n        - id: one-refund\n          type: call_count\n          tool: issue_refund\n          max_calls: 1\n        - id: sane-amount\n          type: arguments\n          tool: issue_refund\n          json_schema:\n            type: object\n            required: [order_id, amount_usd]\n            properties:\n              amount_usd: {type: number, minimum: 0.01, maximum: 500}\n```\n\n- +**Judges that start advisory.**`llm_judge` criteria compare both models' replies pairwise, and`evalshift init` writes them as non-blocking until they are calibrated.\n- +**A verdict decided in advance.** A`migration_policy` block sets the budgets, such as zero\ninvariant violations or at most 10% tool-selection divergence, before anyone sees a number.\n\nOne limit to know: EvalShift does not run your tools against a live system. It feeds each model the\ntool results recorded in production, so it grades the calls a model makes, not the database state\nafter them. Outcome checks against a real environment, like the `refund_issued` check above, belong\nin your own staging tests. The [agents docs](https://www.evalshift.dev/docs/agents) cover replay in detail, and the\n[trace_invariants reference](https://www.evalshift.dev/docs/configuration#trace-invariants) lists every rule type.\n\n## FAQ\n\n**What are AI evals?** Tests for AI systems. An eval gives a model or agent an input, lets it run,\nand grades the result with code, with another model acting as a judge, or with human review. A set\nof evals run together is a suite, and its pass rates show whether a change to the model, prompt or\ntools made the system better or worse.\n\n**What is the difference between LLM evals and agent evals?** An LLM eval grades one response to one\nprompt, usually its text. An agent eval grades a whole run: the tools the agent called, their\norder and arguments, the state the run left behind, and the final reply. Agent evals need graders\non the trace and the outcome, because an agent can write a correct-sounding reply after doing the\nwrong thing.\n\n**How do you write an eval for an AI agent?** Start from 20 to 50 real failures plus a sample of real\ntraffic, define success as an outcome in the world, grade that outcome with code, add trace rules\nonly for steps that must never be skipped, use an LLM judge only for the quality of the reply, run\neach task several times, read the failed transcripts, and freeze the suite so it can gate every\nchange.\n\n**Should agent evals check the tool calls or only the final outcome?** Mostly the outcome, because\nagents often reach a correct result by a path you did not anticipate, and exact-path checks fail\nthose runs. Check the path only for rules that must hold every time, such as verifying before a\nwrite or charging at most once.\n\n**How many eval cases do you need to start?** 20 to 50 cases drawn from real failures is enough to\nstart, because early changes have large effects. To compare two models or prompts per route, aim\nfor 100 to 300 cases from production, stratified so every route you would act on has about 20.\n\n**What is the difference between pass@k and pass^k?** pass@k is the chance that at least one of k\ntrials succeeds, and rises with k. pass^k is the chance that all k trials succeed, and falls with\nk. Use pass^k for customer-facing agents: in τ-bench, GPT-4o solved about 61% of retail tasks on one\ntrial but only about 25% in all of eight trials.\n\n## Keep reading\n\n- +[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) : the four tool-call evaluators,\nand what does not belong in trace expectations.\n- +[LLM-as-a-judge](https://www.evalshift.dev/blog/llm-as-a-judge) : how judges work, how accurate they are and how to\nvalidate one.\n- +[Build a golden eval suite from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic) :\nstep 1 in full.\n- +[How to test an LLM model migration before you ship it](https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship) :\na regression eval for the model change itself.", "url": "https://wpnews.pro/news/ai-evals-for-agents-what-they-are-and-how-to-write-them", "canonical_source": "https://www.evalshift.dev/blog/ai-evals-for-agents", "published_at": "2026-11-02 00:00:00+00:00", "updated_at": "2026-11-02 20:19:57.085475+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "large-language-models", "artificial-intelligence"], "entities": ["Anthropic", "Demystifying evals for AI agents"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-evals-for-agents-what-they-are-and-how-to-write-them", "markdown": "https://wpnews.pro/news/ai-evals-for-agents-what-they-are-and-how-to-write-them.md", "text": "https://wpnews.pro/news/ai-evals-for-agents-what-they-are-and-how-to-write-them.txt", "jsonld": "https://wpnews.pro/news/ai-evals-for-agents-what-they-are-and-how-to-write-them.jsonld"}}