AI evals are tests that grade what a model or agent did. How agent evals differ from LLM evals, the three grader types, and how to write one in 7 steps.
An AI eval is a test for an AI system: you give it an input, let it run, and grade what it did with code, with another model, or with a person. LLM evals grade a single response. Evals for AI agents grade a whole run: the tools the agent called, the arguments it passed, the state it left behind and the reply it ended with. That difference decides how you write them, because an agent can reply "your refund is on its way" without ever calling the refund tool, and an eval that reads only the reply will mark that run as a pass.
This guide defines the parts of an eval, explains how agent evals differ from LLM evals, covers the three kinds of grader, and then walks through writing an eval for an agent in seven steps, with a worked example you can adapt.
What an AI eval is made of #
Anthropic's guide to agent evals (Demystifying evals for AI agents ↗, January 2026) uses a vocabulary most eval tools now share. It is worth adopting, because "the eval failed" can mean any of these:
| Term | What it is | Example for a support agent |
|---|---|---|
| Task | One test case: an input plus what success means | "Order A1042 arrived broken, can I get my money back?" Success: one refund of $59.00 on A1042 |
| Trial | One attempt at a task. Agents vary between runs, so a task gets several | Run 3 of 5 |
| Grader | Logic that scores one aspect of a trial | "Exactly one refund exists for A1042" |
| Transcript (trace) | The full record of a trial: messages, tool calls, tool results, the final reply | check_refund_policy →issue_refund(order_id="A1042", amount_usd=59.0) → reply |
| Outcome | The state of the world after the trial | The row in the refunds table, not the sentence claiming it |
| Suite | The set of tasks you run together | 120 support tasks, tagged by route |
A task can have several graders, and usually should. One that checks the refund exists, one that checks policy was looked up first, and one that checks the reply explains the timeline will each fail for a different reason, and you want to know which.
LLM evals vs agent evals #
The terms overlap and are often used interchangeably, but they grade different things.
| LLM eval | Agent eval | |
|---|---|---|
| Unit graded | One response to one prompt | A run: several model calls, tool calls and results |
| What can go wrong | Wrong, off-topic or malformed text | Wrong tool, wrong arguments, skipped step, repeated action, false claim of success, plus everything an LLM eval catches |
| Where the evidence is | The output text | The trace and the outcome, then the text |
| Determinism | Varies between samples | Varies more: one different tool call changes every step after it |
The practical consequence: an agent's worst regressions read as correct. A model that stops
calling notify_security_team, or reports an address change it never made, still writes fluent,
on-topic prose. Every eval that scores the text alone passes it.
What actually breaks when you switch LLMs goes through
the full change surface.
The three kinds of grader #
Every eval grader is one of three kinds, and a good suite uses all three for different jobs.
| Grader | Examples | Cost | Strength | Weakness |
|---|---|---|---|---|
| Code | Schema validation, exact match, regex, tool-call checks, database state | Free, instant | Deterministic, cannot be argued with | Only checks what you can specify exactly |
| Model (LLM judge) | Rubric scoring, pairwise comparison, natural-language assertions | About $1 per 1,000 calls on a small judge | Handles open-ended text | Biased toward position, length and its own phrasing; needs calibration |
| Human | Expert review, spot checks, labeling | Reviewer hours | The ground truth everything else is calibrated against | Slow, expensive, drifts with fatigue |
The order matters. Use code for everything code can check, a model for the quality of prose that is left, and people to write the criteria, label a calibration sample and read the failures. LLM-as-a-judge covers the model grader in depth, including how accurate judges are and how to validate one.
Capability evals vs regression evals #
Evals answer one of two questions, and mixing them up makes the numbers hard to read.
- +Capability evals ask what the agent can do. They should start with a low pass rate, because a suite the agent already passes cannot show progress. You use them while building.
- +Regression evals ask whether the agent still does what it did yesterday. They should sit near 100%, so any drop is a signal. You run them on every change to the model, the prompt or the tools.
A capability task the agent passes reliably graduates into the regression suite. Most teams building on a model provider spend more of their eval effort on regression than they expect, because the things that change most often are not their code: a model upgrade, a provider switch, a deprecated model with a retirement date.
Agents are not deterministic: pass@k and pass^k #
Run the same agent on the same task five times and you can get five different traces. One trial per task measures luck as much as behavior. Two metrics describe the spread:
- +pass@k: the chance that at least one of k trials succeeds. It rises with k, and it is the right number when one success is enough, as with a coding agent whose output a person will review.
- +pass^k: the chance that all k trials succeed. It falls with k, and it is the right number for an agent customers talk to, because each of them gets one trial.
The gap is large. In τ-bench, a benchmark of agents serving simulated retail and airline customers under written policies, GPT-4o with function calling solved about 61% of retail tasks on a single trial, and its pass^8, solving the same task in all eight trials, fell to about 25% (Yao et al., 2024 ↗). An agent that succeeds 75% of the time per trial passes three trials in a row only 42% of the time (0.75³).
How to write an eval for an AI agent #
The seven steps below take you from nothing to a suite you can gate a release on. They assume an agent in or near production. If you are still prototyping, start with step 1 and stop after step 4.
Step 1: start from real failures and real traffic #
Do not start by imagining test cases. Start with the 20 to 50 cases that already went wrong: bug reports, support escalations, transcripts someone flagged. Anthropic's guide makes the same recommendation, because early on, the changes you make have large effects and a small suite detects them.
Then add real traffic. Hand-written cases cover the situations you already thought of; production logs contain the ones you did not. Sample 100 to 300 inputs, stratified by route or intent so a rare but expensive route is not drowned out, strip personal data, and drop near-duplicates. Build a golden eval suite from production traffic covers capturing those runs, and how many eval cases you need covers sizing.
Tag every case with its route and any known edge case. Pass rates per tag are what you will act on later.
Step 2: define success as an outcome, before you write a grader #
Write down what done looks like for each task in terms of the world, not the reply:
- +"One refund of $59.00 exists on order A1042", not "the agent says the refund was issued".
- +"The booking exists for the 14:00 slot and no other booking was created", not "the reply confirms the booking".
- +"No tool was called" for a question the agent should answer from its instructions.
If you cannot write success down, a grader cannot check it, and neither can a colleague. That is a sign the task is ambiguous, not that you need a cleverer grader. For tasks with a fixed answer, store the expected outcome on the task. For tasks with many valid answers, store the properties every valid answer has.
Step 3: grade the outcome, and the path only where it is a rule #
Agents often reach a correct outcome by a route you did not anticipate: a different lookup order, an extra search, a clarifying question. An eval that requires an exact sequence of tool calls fails those runs and teaches you to ignore failures. Anthropic's advice is to "grade what the agent produced, not the path it took."
There is one exception, and it matters most for agents that act. Some paths are rules, and breaking them is a failure even when the outcome looks right:
- +Authenticate or verify before any write.
- +Look up the refund policy before issuing a refund.
- +Charge a card at most once.
- +Never call a deprecated or dangerous endpoint.
- +Keep an amount inside its allowed bounds.
Write those as checks on the trace, one rule each. Leave everything else free.
Step 4: use code first, a judge for the rest, people to calibrate #
Go through the success definition from step 2 and assign each property to the cheapest grader that can check it.
Code graders take the outcome and the trace rules: the refund row exists, its amount is right,
check_refund_policy ran before issue_refund, issue_refund ran once, the structured output
parses. A sketch in Python, independent of any framework:
def grade(trial, db):
tools = [call.name for call in trial.tool_calls]
refunds = db.refunds_for(order_id="A1042")
refund_issued = len(refunds) == 1 and refunds[0].amount_usd == 59.00
policy_first = (
"check_refund_policy" in tools
and "issue_refund" in tools
and tools.index("check_refund_policy") < tools.index("issue_refund")
)
honest_reply = refund_issued or "refund" not in trial.final_text.lower()
return {
"refund_issued": refund_issued,
"policy_first": policy_first,
"honest_reply": honest_reply,
}
Return one result per check, not one combined score. A task that fails policy_first and passes
refund_issued is a different bug from one that fails both.
A model grader takes what code cannot: whether the reply explains when the money arrives, whether the tone fits an upset customer. Give it one criterion per judge, phrased as observable behavior, with a clear pass condition:
Criterion: Does the reply tell the customer when the refund will reach them?
It passes if it gives a timeframe or a date. It fails if it says only that
the refund was issued, or gives no timing at all.
Answer PASS or FAIL, then one sentence of reasoning.
People label 30 to 50 trials by hand and compare their labels with the judge's. Every disagreement is either a vague criterion or a judge you should not trust yet. Keep a new judge advisory, reported but not blocking, until that comparison holds up.
Step 5: run every task several times #
Run each task at least three times, five where the stakes are high, and report pass^k for customer-facing behavior: the share of tasks the agent got right every time. A task that passes two runs out of three is not a pass. It is a flaky behavior your customers will meet one time in three.
Several trials also separate noise from regressions. If the current model already disagrees with itself on a task, a different result from a new model on that task proves nothing on its own.
Step 6: read the transcripts, not just the score #
An aggregate pass rate tells you that something moved, not what. Break results down per tag and per grader, and read every failed transcript in the routes you care about. Most eval bugs show up there first: a grader checking the wrong field, a task with an impossible success condition, a test case that leaked into the prompt.
Read some passes too. A grader that never fails is either checking something trivial or broken.
Step 7: freeze the suite and run it on every change #
A regression suite only proves something if it holds still. Freeze the tasks and graders, version them, and run the suite on every change that can alter behavior: the model name, the prompt, the tool definitions, the agent's code. Comparing two runs on a suite that changed in between compares two moving things.
When a change is intended, such as a new route or a corrected expectation, update the suite in its own commit, so the next comparison starts from a known baseline. Then wire the run into CI, so a pull request that regresses the agent fails before it merges. LLM regression testing in CI shows a gate on GitHub Actions.
Common mistakes in agent evals #
- +Grading the claim instead of the outcome. "The agent said it booked the slot" is not a booking. Check the state, or at least the tool call that would have changed it.
- +Requiring an exact path. Pinning every tool call in order fails valid runs. Pin only the rules from step 3.
- +One averaged score. "Quality dropped 3%" cannot be fixed. "Twelve tasks on the refund route
skipped
check_refund_policy" can. - +One trial per task. You will chase noise and miss flakiness.
- +A judge nobody validated. A judge returns a confident number whether or not its criterion makes sense. Calibrate it against human labels first.
- +Changing two things at once. A new model and a rewritten prompt in one comparison leave nobody able to say which one moved the score. Change one, measure, then change the other.
Writing agent evals with EvalShift #
EvalShift is built for the regression side of agent evals, and for the change most teams test least: switching or upgrading the model. It replays a frozen suite against the current model and a candidate, grades both sides of every task, and turns the differences into a verdict.
- +Tasks from real traffic. The capture SDK records production runs, and
evalshift capture syncturns them into a golden suite where every task carries its recorded tools and expected tool calls. - +Trace graders, no code required.
tool_selectionandtool_argumentscheck which tools were called and with which values, andtrace_invariantsholds both models to the rules from step 3:
evaluators:
trace_invariants:
- name: refunds_contract
owner: "@support-team"
rules:
- id: policy-before-refund
type: order
before: check_refund_policy
after: issue_refund
- id: one-refund
type: call_count
tool: issue_refund
max_calls: 1
- id: sane-amount
type: arguments
tool: issue_refund
json_schema:
type: object
required: [order_id, amount_usd]
properties:
amount_usd: {type: number, minimum: 0.01, maximum: 500}
- +Judges that start advisory.
llm_judgecriteria compare both models' replies pairwise, andevalshift initwrites them as non-blocking until they are calibrated. - +A verdict decided in advance. A
migration_policyblock sets the budgets, such as zero invariant violations or at most 10% tool-selection divergence, before anyone sees a number.
One limit to know: EvalShift does not run your tools against a live system. It feeds each model the
tool results recorded in production, so it grades the calls a model makes, not the database state
after them. Outcome checks against a real environment, like the refund_issued check above, belong
in your own staging tests. The agents docs cover replay in detail, and the
trace_invariants reference lists every rule type.
FAQ #
What are AI evals? Tests for AI systems. An eval gives a model or agent an input, lets it run, and grades the result with code, with another model acting as a judge, or with human review. A set of evals run together is a suite, and its pass rates show whether a change to the model, prompt or tools made the system better or worse.
What is the difference between LLM evals and agent evals? An LLM eval grades one response to one prompt, usually its text. An agent eval grades a whole run: the tools the agent called, their order and arguments, the state the run left behind, and the final reply. Agent evals need graders on the trace and the outcome, because an agent can write a correct-sounding reply after doing the wrong thing.
How do you write an eval for an AI agent? Start from 20 to 50 real failures plus a sample of real traffic, define success as an outcome in the world, grade that outcome with code, add trace rules only for steps that must never be skipped, use an LLM judge only for the quality of the reply, run each task several times, read the failed transcripts, and freeze the suite so it can gate every change.
Should agent evals check the tool calls or only the final outcome? Mostly the outcome, because agents often reach a correct result by a path you did not anticipate, and exact-path checks fail those runs. Check the path only for rules that must hold every time, such as verifying before a write or charging at most once.
How many eval cases do you need to start? 20 to 50 cases drawn from real failures is enough to start, because early changes have large effects. To compare two models or prompts per route, aim for 100 to 300 cases from production, stratified so every route you would act on has about 20.
What is the difference between pass@k and pass^k? pass@k is the chance that at least one of k trials succeeds, and rises with k. pass^k is the chance that all k trials succeed, and falls with k. Use pass^k for customer-facing agents: in τ-bench, GPT-4o solved about 61% of retail tasks on one trial but only about 25% in all of eight trials.
Keep reading #
- +Evaluating agent tool calls : the four tool-call evaluators, and what does not belong in trace expectations.
- +LLM-as-a-judge : how judges work, how accurate they are and how to validate one.
- +Build a golden eval suite from production traffic : step 1 in full.
- +How to test an LLM model migration before you ship it : a regression eval for the model change itself.