How to Build an Evaluation Harness for Your AI Agent (So It Doesn't Break in Production)

A developer built an evaluation harness for AI agents to catch silent regressions and tool-call errors that manual testing misses. The harness uses four graders—Truth, Path, Judge, and Gate—to check tool arguments, trajectory efficiency, goal completion via LLM-as-judge, and aggregate regression detection. The pattern works with any agent framework and integrates with CI to fail builds when changes degrade agent performance.

The agent passed every test I threw at it by hand. Then a user asked it to "book the cheaper flight," it happily called book on the wrong flight ID, and nobody caught it for three days because the demo I kept running never once asked for the cheaper flight. That is the trap. When you test an agent by running it yourself, the demo is the test — you type the happy-path prompt, watch it work, and ship. But your users don't type your happy path, and your model provider silently ships a new checkpoint next Tuesday. An AI agent evaluation harness is the thing that tells you your agent actually works, and — more importantly — that last week's prompt tweak didn't quietly break the flight-booking flow you're not looking at. This tutorial builds one. By the end you'll have a small, runnable Python repo that grades an agent on real tasks, checks that it called the right tools with the right arguments, judges the fuzzy stuff with an LLM, and fails your CI build when a change makes the agent worse . No product pitch, no framework lock-in — the pattern works with LangChain, LangGraph, raw MCP, or whatever you wired together yourself. We'll test a tiny travel-support agent with three tools — search , get weather , and book — because tool calls are checkable in a way that free-form chat isn't. Along the way we'll build four graders. I call them the Four Graders , and they map cleanly onto what you're actually afraid of: | Grader | Question it answers | How it works | |---|---|---| Truth | Did the agent call the right tool with the right arguments? | Deterministic code check | Path | Did it take a sane route, or loop and thrash? | Trajectory / step-efficiency check | Judge | Did it actually accomplish the goal when there's no single right string? | LLM-as-judge with a rubric | Gate | Did this change make the agent worse than last week? | Aggregate score + CI regression gate | Truth and Path are cheap, fast, and deterministic. Judge is flexible but needs calibration. Gate is what turns all of it into a safety net instead of a science project. pip install . pytest pip install pytest . For the LLM-as-judge step, any provider SDK — we'll use openai as the example, but the call is one swappable function, so Anthropic, a local model, or anything with a chat endpoint works.Three things make agents uniquely dangerous to eyeball-test. The demo is the test. You built the agent by running prompts and fixing what broke. So the prompts you run are, by construction, the ones it already handles. You are grading the exam you wrote the answer key for. Non-determinism. Same prompt, same model, different output. Temperature, a re-ranked retrieval, a provider-side model update — any of these can flip a passing run into a failing one tomorrow with zero code change on your side. A single manual run tells you almost nothing about the distribution of behavior. Silent regressions. This is the killer. You tighten a system prompt to fix formatting, and it turns out that phrasing also nudged the model to skip the get weather check before booking. Nothing errors. Nothing logs red. The agent just gets 8% worse at a task you weren't watching, and you find out from a user. An eval harness is a smoke detector for exactly this. The fix isn't "test more by hand." It's a small set of tasks, graded automatically, run on every change. Before we grade anything, know where you're grading. An agent run is a tree of spans — the top-level task, the reasoning steps and tool calls underneath, and the individual components a retriever, a single tool, a sub-agent at the leaves. You can evaluate at three altitudes, and good harnesses use all three Confident AI https://www.confident-ai.com/blog/llm-agent-evaluation-complete-guide lays these out well : Start end-to-end and trajectory. Drop to component-level only when you need to localize a failure. Here's the mapping onto a span tree: end-to-end Task: "Book AA123 Fri unless it's raining in Denver" │ trajectory ├─ get weather city="Denver" ← Path + Truth grade here ├─ book flight id="AA123", ... ← Truth grades args here └─ final answer: "Booked AA123..." ← Judge grades here │ component └─ weather tool latency / retriever recall ← only when localizing Everything downstream depends on this. Your dataset is a small set of tasks with known-good expectations. Two rules, both from Anthropic's eval guide https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents : Here's a starter dataset for our travel agent. This is a plain Python list — put it in evals/dataset.py : evals/dataset.py — the golden set. Each row is one checkable task. GOLDEN = { "task id": "weather-gate-book", "input": "Book flight AA123 for Friday, but only if it isn't raining in Denver.", "expect tools": {"name": "get weather", "args": {"city": "Denver"}}, {"name": "book", "args": {"flight id": "AA123", "date": "Friday"}}, , "success contains": "booked", }, { "task id": "search-before-book", "input": "Find the cheapest flight to Seattle next Monday and book it.", "expect tools": {"name": "search", "args": {"destination": "Seattle"}}, {"name": "book", "args": {}}, empty = don't care about specific args , "success contains": "booked", }, { "task id": "no-book-when-raining", "input": "Book AA200 Friday only if Denver is clear. It is raining. ", "expect tools": {"name": "get weather", "args": {"city": "Denver"}} , "forbid tools": "book" , booking here is a hard failure "success contains": "did not book", }, { "task id": "weather-only", "input": "What's the weather in Denver?", "expect tools": {"name": "get weather", "args": {"city": "Denver"}} , "forbid tools": "book", "search" , "success contains": "denver", }, { "task id": "ambiguous-needs-search", "input": "Book me something cheap to the coast.", "expect tools": {"name": "search", "args": {}} , "success contains": "options", should clarify/search, not blind-book }, What you should see: five tasks, each with expected tool calls, some with forbid tools calls that must not happen and a success contains string for the final answer. Notice no-book-when-raining and ambiguous-needs-search — those are the sneaky-failure cases a happy-path demo never exercises. That's the whole point. We also need a uniform shape for what the agent did . Put this in evals/types.py : python evals/types.py — the trace shape every grader reads. from dataclasses import dataclass @dataclass class ToolCall: name: str args: dict @dataclass class AgentRun: task id: str tool calls: list list ToolCall , in order final answer: str Your job when integrating: make your real agent emit an AgentRun . LangChain/LangGraph expose this through callbacks or the run tree; MCP servers log tool invocations; a hand-rolled loop already has the list. For the tutorial, we'll grade sample runs directly so every block below is runnable without an API key. Truth answers the cheapest, highest-value question: did the agent call the right tools with the right arguments? This is a code-based grader — fast, free, objective, reproducible. No LLM involved. Put it in evals/graders.py : evals/graders.py — Truth: are the required tool calls present & correct? def grade truth run, task : actual = run.tool calls 1. Forbidden calls are a hard fail. for c in actual: if c.name in task.get "forbid tools", : return 0.0, f"called forbidden tool: {c.name}" 2. Every expected call must appear with matching args. for exp in task "expect tools" : hit = next c for c in actual if c.name == exp "name" and all c.args.get k == v for k, v in exp "args" .items , None, if hit is None: return 0.0, f"missing/incorrect: {exp 'name' } {exp 'args' } " return 1.0, "all expected tool calls present with correct args" Two distinct checks are hiding in there, and it's worth naming them because the eval literature does: tool correctness was the right tool called at all? and argument correctness were the parameters right? . An agent that calls book — correct tool — with the wrong flight id — wrong argument — is a bug that costs real money. grade truth catches both. An empty args dict means "I only care that the tool was called, not how," which is how you keep tests from being brittle. Let's run it on a good run and a buggy one: try truth.py — runnable: grade a good run and a broken one. from evals.types import AgentRun, ToolCall from evals.dataset import GOLDEN from evals.graders import grade truth task = GOLDEN 0 weather-gate-book good = AgentRun "weather-gate-book", ToolCall "get weather", {"city": "Denver"} , ToolCall "book", {"flight id": "AA123", "date": "Friday"} , "Booked AA123 for Friday." buggy = AgentRun "weather-gate-book", ToolCall "get weather", {"city": "Denver"} , ToolCall "book", {"flight id": "AA999", "date": "Friday"} , wrong ID "Booked AA999 for Friday." print grade truth good, task print grade truth buggy, task What you should see: 1.0, 'all expected tool calls present with correct args' 0.0, "missing/incorrect: book {'flight id': 'AA123', 'date': 'Friday'} " The buggy run booked the wrong flight and the grader caught it deterministically, in milliseconds, for zero cost. This one grader would have saved me those three days. Truth tells you the agent eventually did the right things. Path tells you it didn't flail to get there. An agent that calls search six times, loops on get weather , and burns 40 seconds and $0.30 to book one flight is a production incident waiting to happen — slow, expensive, and one retry-loop away from a runaway bill. The metric here is step efficiency : did it avoid unnecessary steps, retries, and loops? Add to evals/graders.py : evals/graders.py cont. — Path: efficient trajectory, no thrashing. def grade path run, task : expected = len task "expect tools" actual = len run.tool calls Duplicate identical calls = looping/thrashing. seen, duplicates = set , 0 for c in run.tool calls: key = c.name, tuple sorted c.args.items if key in seen: duplicates += 1 seen.add key if duplicates: return 0.0, f"{duplicates} duplicate tool call s — looping" if actual expected 2: generous budget: 2x the minimal path return 0.0, f"inefficient: {actual} calls for a {expected}-call task" return 1.0, f"efficient: {actual} calls, no duplicates" The expected 2 budget is deliberately generous — real agents legitimately take an extra step to clarify or recover. You're not demanding the minimal path, you're catching pathological paths. Tune the multiplier to your agent; the duplicate-call check is the part that catches infinite-loop regressions before your bill does. Run it against a thrashing agent: grade a run that loops on the same weather call twice from evals.types import AgentRun, ToolCall from evals.dataset import GOLDEN from evals.graders import grade path looper = AgentRun "weather-gate-book", ToolCall "get weather", {"city": "Denver"} , ToolCall "get weather", {"city": "Denver"} , duplicate ToolCall "book", {"flight id": "AA123", "date": "Friday"} , "Booked AA123." print grade path looper, GOLDEN 0 What you should see: 0.0, '1 duplicate tool call s — looping' Some things you can't check with an == . "Did the agent's final answer actually accomplish the user's goal?" often has no single correct string. That's where a model-based grader earns its keep: flexible, handles open-ended tasks, scores against a rubric. The cost is that it's non-deterministic and — read this twice — must be calibrated against human labels before you trust it . More on that caveat below. The key design move: keep the LLM call in one swappable function so the grader isn't married to any provider. Add to evals/graders.py : evals/graders.py cont. — Judge: LLM scores goal completion via a rubric. import json RUBRIC = """You are a strict grader for an AI agent's answer. Score 1 only if the answer fully accomplishes the user's goal; else 0. User task: {task} Agent's final answer: {answer} Respond with ONLY a JSON object: {{"score": 0 or 1, "reason": "<one sentence "}}""" def grade judge run, task, chat : prompt = RUBRIC.format task=task "input" , answer=run.final answer data = json.loads chat prompt chat is provider-agnostic return float data "score" , data "reason" chat is any function that takes a prompt string and returns the model's text. Here's the OpenAI version — swap the body for Anthropic, Gemini, or a local model and nothing else changes: evals/judge client.py — one swappable function. Works with any provider. def openai chat prompt: str - str: from openai import OpenAI client = OpenAI reads OPENAI API KEY from env resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": prompt} , temperature=0, determinism where you can get it response format={"type": "json object"}, return resp.choices 0 .message.content What you should see when you call grade judge run, task, openai chat : a tuple like 1.0, 'The agent booked the requested flight after checking weather.' . Set temperature=0 and force JSON output — you want the judge as reproducible as a stochastic thing can be. Now the caveat I promised, because skipping it is how teams fool themselves: an unvalidated LLM judge is a vibe with a number attached. Before you trust it in CI, hand-label 20–30 runs yourself, run the judge on the same runs, and check agreement. If the judge disagrees with you often, fix the rubric — don't ship the judge. Anthropic is blunt about this: model-based graders "require calibration against human judgment," and human labels are the gold standard you calibrate against. Use the judge for the fuzzy end-to-end verdict; lean on Truth and Path — which need no calibration — for everything you can check deterministically. Here's where it stops being a script and becomes a safety net. The Gate does two jobs: run every grader over every task, and fail the build when the aggregate score drops below a baseline . That last part is the entire reason to do any of this — it's what catches the silent regression. One more principle before the code: isolate your grading . Anthropic's rule is that "each trial should be 'isolated' by starting from a clean environment" — no shared state bleeding between runs and inflating or tanking your scores. In practice: fresh fixtures per task, no reused sessions, no test depending on the one before it. pytest gives you this for free if you don't fight it. Put this in evals/test agent.py : python evals/test agent.py — the gate. Runs in CI on every PR. import pytest from evals.dataset import GOLDEN from evals.graders import grade truth, grade path from evals.runner import run agent your agent → AgentRun DETERMINISTIC = {"truth": grade truth, "path": grade path} BASELINE = 0.90 ship bar: aggregate score must stay = this @pytest.mark.parametrize "task", GOLDEN, ids=lambda t: t "task id" def test each task passes deterministic graders task : run = run agent task "input" clean run per task = isolation for name, grader in DETERMINISTIC.items : score, reason = grader run, task assert score == 1.0, f" {name} {task 'task id' }: {reason}" def test aggregate no regression : scores = for task in GOLDEN: run = run agent task "input" per task = g run, task 0 for g in DETERMINISTIC.values scores.append sum per task / len per task avg = sum scores / len scores assert avg = BASELINE, f"REGRESSION: {avg:.2f} < baseline {BASELINE}" What you should see: run pytest evals/ -q . Each golden task shows up as a named test weather-gate-book , no-book-when-raining , … , and a failing task prints exactly which grader failed and why — truth weather-gate-book: missing/incorrect: book {'flight id': 'AA123'...} . Break your prompt on purpose and watch the aggregate test go red with the score. That red X is the thing you were missing. I kept the Judge grader out of the hard assert gate on purpose: because it's non-deterministic, treat its score as a monitored metric log it, alert on drops rather than a pass/fail that flakes your build. Deterministic graders gate; the judge informs. Now wire it into CI so it runs on every pull request. GitHub Actions, .github/workflows/agent-evals.yml : .github/workflows/agent-evals.yml — block the merge on a regression. name: agent-evals on: pull request jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install pytest openai - run: pytest evals/ -q env: OPENAI API KEY: ${{ secrets.OPENAI API KEY }} What you should see: open a PR that changes the prompt or bumps the model, and the agent-evals check runs. If the agent got worse, the check is red and the merge is blocked — same as a failing unit test. Your agent is now gated in CI like normal software, which is the whole point. You now have the four pieces that separate "it worked in the demo" from "I know it works": a golden dataset drawn from real failures, a Truth grader that catches wrong tools and wrong arguments deterministically, a Path grader that catches looping and thrash, a Judge for the fuzzy goal-completion verdict, and a Gate that fails your build the moment a change makes the agent worse. None of it is exotic. It's the same instinct that made you write unit tests for the rest of your code — you just didn't have a shape for it when the thing under test was a non-deterministic agent calling tools. Now you do. Where to go next: Your agent looks great when you run it by hand. That was never the question. The question is whether it still works after the next prompt tweak, the next model update, and the next user who does something you didn't demo. Build the harness, and you'll actually know.