You tweak one line of your agent's prompt to fix a complaint. The fix works. You ship it.
Two days later, a different user reports something that used to work just fine.
Nobody touched that feature. You changed the prompt, and the agent's behavior shifted everywhere at once. That's why AI agents need evals, and why the evals most teams write miss the bugs that matter.
TL;DR
An eval gives the agent a task, runs it, and scores the result automatically. Here's one for a scheduling agent:
task: "Book 30 minutes with Priya next Tuesday afternoon"
expect:
- create_event called once
- attendee is priya@example.com
- start time is Tuesday, between 12:00 and 17:00
- no existing meeting was moved or deleted
trials: 5
pass: all 5 trials
Run your suite after every prompt, model, or tool change, and you'll know what broke before your users do.
Most eval suites only check the final reply. But an agent can say "Booked!" while calling the wrong tool, or call the right tool with the wrong date. Check all three: the outcome, the trajectory (tool calls and their arguments), and the state that changed.
Start with plain code. It's fast, free, and never disagrees with itself:
def grade(trace, calendar):
events = [c for c in trace.tool_calls if c.name == "create_event"]
assert len(events) == 1
assert "priya@example.com" in events[0].args["attendees"]
assert calendar.unchanged_except(events[0].result_id)
Use an LLM judge only for things code can't check, like tone or whether an answer is actually helpful. Then have a human review a sample of the judge's scores now and then, so you know the judge is grading the way you would.
Agents are non-deterministic. If a task passes 90% of the time, the chance it passes five times in a row is only 59%. For anything user-facing, "it worked when I tried it" isn't good enough. Measure how often it works every time.
Don't invent test cases. Mine them:
Twenty real cases beat two hundred made-up ones.
Regression evals check things that already work. They should pass almost 100% of the time, and a failure blocks the release.
Capability evals check things the agent can't do reliably yet. A low score is expected. Watching it climb tells you whether your changes are actually helping.
An eval suite is the agent's memory of every mistake it has made. Without one, every prompt change is a guess.
How do you decide when an agent change is safe to ship? Share in the comments.