The agent passed every test I threw at it by hand. Then a user asked it to "book the cheaper flight," it happily called book
on the wrong flight ID, and nobody caught it for three days because the demo I kept running never once asked for the cheaper flight.
That is the trap. When you test an agent by running it yourself, the demo is the test — you type the happy-path prompt, watch it work, and ship. But your users don't type your happy path, and your model provider silently ships a new checkpoint next Tuesday. An AI agent evaluation harness is the thing that tells you your agent actually works, and — more importantly — that last week's prompt tweak didn't quietly break the flight-booking flow you're not looking at.
This tutorial builds one. By the end you'll have a small, runnable Python repo that grades an agent on real tasks, checks that it called the right tools with the right arguments, judges the fuzzy stuff with an LLM, and fails your CI build when a change makes the agent worse. No product pitch, no framework lock-in — the pattern works with LangChain, LangGraph, raw MCP, or whatever you wired together yourself.
We'll test a tiny travel-support agent with three tools — search
, get_weather
, and book
— because tool calls are checkable in a way that free-form chat isn't. Along the way we'll build four graders. I call them the Four Graders, and they map cleanly onto what you're actually afraid of:
| Grader | Question it answers | How it works |
|---|---|---|
| Truth | ||
| Did the agent call the right tool with the right arguments? | Deterministic code check | |
| Path | ||
| Did it take a sane route, or loop and thrash? | Trajectory / step-efficiency check | |
| Judge | ||
| Did it actually accomplish the goal when there's no single right string? | LLM-as-judge with a rubric | |
| Gate | ||
| Did this change make the agent worse than last week? | ||
| Aggregate score + CI regression gate |
Truth and Path are cheap, fast, and deterministic. Judge is flexible but needs calibration. Gate is what turns all of it into a safety net instead of a science project.
pip install
.pytest
(pip install pytest
). For the LLM-as-judge step, any provider SDK — we'll use openai
as the example, but the call is one swappable function, so Anthropic, a local model, or anything with a chat endpoint works.Three things make agents uniquely dangerous to eyeball-test.
The demo is the test. You built the agent by running prompts and fixing what broke. So the prompts you run are, by construction, the ones it already handles. You are grading the exam you wrote the answer key for.
Non-determinism. Same prompt, same model, different output. Temperature, a re-ranked retrieval, a provider-side model update — any of these can flip a passing run into a failing one tomorrow with zero code change on your side. A single manual run tells you almost nothing about the distribution of behavior.
Silent regressions. This is the killer. You tighten a system prompt to fix formatting, and it turns out that phrasing also nudged the model to skip the get_weather
check before booking. Nothing errors. Nothing logs red. The agent just gets 8% worse at a task you weren't watching, and you find out from a user. An eval harness is a smoke detector for exactly this.
The fix isn't "test more by hand." It's a small set of tasks, graded automatically, run on every change.
Before we grade anything, know where you're grading. An agent run is a tree of spans — the top-level task, the reasoning steps and tool calls underneath, and the individual components (a retriever, a single tool, a sub-agent) at the leaves. You can evaluate at three altitudes, and good harnesses use all three (Confident AI lays these out well):
Start end-to-end and trajectory. Drop to component-level only when you need to localize a failure. Here's the mapping onto a span tree:
[end-to-end] Task: "Book AA123 Fri unless it's raining in Denver"
│
[trajectory] ├─ get_weather(city="Denver") ← Path + Truth grade here
├─ book(flight_id="AA123", ...) ← Truth grades args here
└─ final_answer: "Booked AA123..." ← Judge grades here
│
[component] └─ weather_tool latency / retriever recall ← only when localizing
Everything downstream depends on this. Your dataset is a small set of tasks with known-good expectations. Two rules, both from Anthropic's eval guide:
Here's a starter dataset for our travel agent. This is a plain Python list — put it in evals/dataset.py
:
GOLDEN = [
{
"task_id": "weather-gate-book",
"input": "Book flight AA123 for Friday, but only if it isn't raining in Denver.",
"expect_tools": [
{"name": "get_weather", "args": {"city": "Denver"}},
{"name": "book", "args": {"flight_id": "AA123", "date": "Friday"}},
],
"success_contains": "booked",
},
{
"task_id": "search-before-book",
"input": "Find the cheapest flight to Seattle next Monday and book it.",
"expect_tools": [
{"name": "search", "args": {"destination": "Seattle"}},
{"name": "book", "args": {}}, # empty = don't care about specific args
],
"success_contains": "booked",
},
{
"task_id": "no-book-when-raining",
"input": "Book AA200 Friday only if Denver is clear. (It is raining.)",
"expect_tools": [{"name": "get_weather", "args": {"city": "Denver"}}],
"forbid_tools": ["book"], # booking here is a hard failure
"success_contains": "did not book",
},
{
"task_id": "weather-only",
"input": "What's the weather in Denver?",
"expect_tools": [{"name": "get_weather", "args": {"city": "Denver"}}],
"forbid_tools": ["book", "search"],
"success_contains": "denver",
},
{
"task_id": "ambiguous-needs-search",
"input": "Book me something cheap to the coast.",
"expect_tools": [{"name": "search", "args": {}}],
"success_contains": "options", # should clarify/search, not blind-book
},
]
What you should see: five tasks, each with expected tool calls, some with forbid_tools
(calls that must not happen) and a success_contains
string for the final answer. Notice no-book-when-raining
and ambiguous-needs-search
— those are the sneaky-failure cases a happy-path demo never exercises. That's the whole point.
We also need a uniform shape for what the agent did. Put this in evals/types.py
:
from dataclasses import dataclass
@dataclass
class ToolCall:
name: str
args: dict
@dataclass
class AgentRun:
task_id: str
tool_calls: list # list[ToolCall], in order
final_answer: str
Your job when integrating: make your real agent emit an AgentRun
. LangChain/LangGraph expose this through callbacks or the run tree; MCP servers log tool invocations; a hand-rolled loop already has the list. For the tutorial, we'll grade sample runs directly so every block below is runnable without an API key.
Truth answers the cheapest, highest-value question: did the agent call the right tools with the right arguments? This is a code-based grader — fast, free, objective, reproducible. No LLM involved. Put it in evals/graders.py
:
def grade_truth(run, task):
actual = run.tool_calls
for c in actual:
if c.name in task.get("forbid_tools", []):
return 0.0, f"called forbidden tool: {c.name}"
for exp in task["expect_tools"]:
hit = next(
(c for c in actual
if c.name == exp["name"]
and all(c.args.get(k) == v for k, v in exp["args"].items())),
None,
)
if hit is None:
return 0.0, f"missing/incorrect: {exp['name']}({exp['args']})"
return 1.0, "all expected tool calls present with correct args"
Two distinct checks are hiding in there, and it's worth naming them because the eval literature does: tool correctness (was the right tool called at all?) and argument correctness (were the parameters right?). An agent that calls book
— correct tool — with the wrong flight_id
— wrong argument — is a bug that costs real money. grade_truth
catches both. An empty args
dict means "I only care that the tool was called, not how," which is how you keep tests from being brittle.
Let's run it on a good run and a buggy one:
from evals.types import AgentRun, ToolCall
from evals.dataset import GOLDEN
from evals.graders import grade_truth
task = GOLDEN[0] # weather-gate-book
good = AgentRun("weather-gate-book",
[ToolCall("get_weather", {"city": "Denver"}),
ToolCall("book", {"flight_id": "AA123", "date": "Friday"})],
"Booked AA123 for Friday.")
buggy = AgentRun("weather-gate-book",
[ToolCall("get_weather", {"city": "Denver"}),
ToolCall("book", {"flight_id": "AA999", "date": "Friday"})], # wrong ID
"Booked AA999 for Friday.")
print(grade_truth(good, task))
print(grade_truth(buggy, task))
What you should see:
(1.0, 'all expected tool calls present with correct args')
(0.0, "missing/incorrect: book({'flight_id': 'AA123', 'date': 'Friday'})")
The buggy run booked the wrong flight and the grader caught it deterministically, in milliseconds, for zero cost. This one grader would have saved me those three days.
Truth tells you the agent eventually did the right things. Path tells you it didn't flail to get there. An agent that calls search
six times, loops on get_weather
, and burns 40 seconds and $0.30 to book one flight is a production incident waiting to happen — slow, expensive, and one retry-loop away from a runaway bill. The metric here is step efficiency: did it avoid unnecessary steps, retries, and loops?
Add to evals/graders.py
:
def grade_path(run, task):
expected = len(task["expect_tools"])
actual = len(run.tool_calls)
seen, duplicates = set(), 0
for c in run.tool_calls:
key = (c.name, tuple(sorted(c.args.items())))
if key in seen:
duplicates += 1
seen.add(key)
if duplicates:
return 0.0, f"{duplicates} duplicate tool call(s) — looping"
if actual > expected * 2: # generous budget: 2x the minimal path
return 0.0, f"inefficient: {actual} calls for a {expected}-call task"
return 1.0, f"efficient: {actual} calls, no duplicates"
The expected * 2
budget is deliberately generous — real agents legitimately take an extra step to clarify or recover. You're not demanding the minimal path, you're catching pathological paths. Tune the multiplier to your agent; the duplicate-call check is the part that catches infinite-loop regressions before your bill does.
Run it against a thrashing agent:
from evals.types import AgentRun, ToolCall
from evals.dataset import GOLDEN
from evals.graders import grade_path
looper = AgentRun("weather-gate-book",
[ToolCall("get_weather", {"city": "Denver"}),
ToolCall("get_weather", {"city": "Denver"}), # duplicate
ToolCall("book", {"flight_id": "AA123", "date": "Friday"})],
"Booked AA123.")
print(grade_path(looper, GOLDEN[0]))
What you should see:
(0.0, '1 duplicate tool call(s) — looping')
Some things you can't check with an ==
. "Did the agent's final answer actually accomplish the user's goal?" often has no single correct string. That's where a model-based grader earns its keep: flexible, handles open-ended tasks, scores against a rubric. The cost is that it's non-deterministic and — read this twice — must be calibrated against human labels before you trust it. More on that caveat below.
The key design move: keep the LLM call in one swappable function so the grader isn't married to any provider. Add to evals/graders.py
:
import json
RUBRIC = """You are a strict grader for an AI agent's answer.
Score 1 only if the answer fully accomplishes the user's goal; else 0.
User task: {task}
Agent's final answer: {answer}
Respond with ONLY a JSON object: {{"score": 0 or 1, "reason": "<one sentence>"}}"""
def grade_judge(run, task, chat):
prompt = RUBRIC.format(task=task["input"], answer=run.final_answer)
data = json.loads(chat(prompt)) # chat() is provider-agnostic
return float(data["score"]), data["reason"]
chat
is any function that takes a prompt string and returns the model's text. Here's the OpenAI version — swap the body for Anthropic, Gemini, or a local model and nothing else changes:
def openai_chat(prompt: str) -> str:
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY from env
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0, # determinism where you can get it
response_format={"type": "json_object"},
)
return resp.choices[0].message.content
What you should see when you call grade_judge(run, task, openai_chat)
: a tuple like (1.0, 'The agent booked the requested flight after checking weather.')
. Set temperature=0
and force JSON output — you want the judge as reproducible as a stochastic thing can be.
Now the caveat I promised, because skipping it is how teams fool themselves: an unvalidated LLM judge is a vibe with a number attached. Before you trust it in CI, hand-label 20–30 runs yourself, run the judge on the same runs, and check agreement. If the judge disagrees with you often, fix the rubric — don't ship the judge. Anthropic is blunt about this: model-based graders "require calibration against human judgment," and human labels are the gold standard you calibrate against. Use the judge for the fuzzy end-to-end verdict; lean on Truth and Path — which need no calibration — for everything you can check deterministically.
Here's where it stops being a script and becomes a safety net. The Gate does two jobs: run every grader over every task, and fail the build when the aggregate score drops below a baseline. That last part is the entire reason to do any of this — it's what catches the silent regression.
One more principle before the code: isolate your grading. Anthropic's rule is that "each trial should be 'isolated' by starting from a clean environment" — no shared state bleeding between runs and inflating (or tanking) your scores. In practice: fresh fixtures per task, no reused sessions, no test depending on the one before it. pytest
gives you this for free if you don't fight it.
Put this in evals/test_agent.py
:
import pytest
from evals.dataset import GOLDEN
from evals.graders import grade_truth, grade_path
from evals.runner import run_agent # your agent → AgentRun
DETERMINISTIC = {"truth": grade_truth, "path": grade_path}
BASELINE = 0.90 # ship bar: aggregate score must stay >= this
@pytest.mark.parametrize("task", GOLDEN, ids=lambda t: t["task_id"])
def test_each_task_passes_deterministic_graders(task):
run = run_agent(task["input"]) # clean run per task = isolation
for name, grader in DETERMINISTIC.items():
score, reason = grader(run, task)
assert score == 1.0, f"[{name}] {task['task_id']}: {reason}"
def test_aggregate_no_regression():
scores = []
for task in GOLDEN:
run = run_agent(task["input"])
per_task = [g(run, task)[0] for g in DETERMINISTIC.values()]
scores.append(sum(per_task) / len(per_task))
avg = sum(scores) / len(scores)
assert avg >= BASELINE, f"REGRESSION: {avg:.2f} < baseline {BASELINE}"
What you should see: run pytest evals/ -q
. Each golden task shows up as a named test (weather-gate-book
, no-book-when-raining
, …), and a failing task prints exactly which grader failed and why — [truth] weather-gate-book: missing/incorrect: book({'flight_id': 'AA123'...})
. Break your prompt on purpose and watch the aggregate test go red with the score. That red X is the thing you were missing.
I kept the Judge grader out of the hard assert
gate on purpose: because it's non-deterministic, treat its score as a monitored metric (log it, alert on drops) rather than a pass/fail that flakes your build. Deterministic graders gate; the judge informs.
Now wire it into CI so it runs on every pull request. GitHub Actions, .github/workflows/agent-evals.yml
:
name: agent-evals
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install pytest openai
- run: pytest evals/ -q
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
What you should see: open a PR that changes the prompt or bumps the model, and the agent-evals
check runs. If the agent got worse, the check is red and the merge is blocked — same as a failing unit test. Your agent is now gated in CI like normal software, which is the whole point.
You now have the four pieces that separate "it worked in the demo" from "I know it works": a golden dataset drawn from real failures, a Truth grader that catches wrong tools and wrong arguments deterministically, a Path grader that catches looping and thrash, a Judge for the fuzzy goal-completion verdict, and a Gate that fails your build the moment a change makes the agent worse.
None of it is exotic. It's the same instinct that made you write unit tests for the rest of your code — you just didn't have a shape for it when the thing under test was a non-deterministic agent calling tools. Now you do.
Where to go next:
Your agent looks great when you run it by hand. That was never the question. The question is whether it still works after the next prompt tweak, the next model update, and the next user who does something you didn't demo. Build the harness, and you'll actually know.