A unit test fails the same way every time. An agent can fail five different ways, pass on the sixth attempt, and take a path nobody expected on the seventh.
That does not make traditional testing obsolete. It makes the old testing pyramid useful again, with a few new layers and a different idea of what a pass means.
I spent much of the first half of my engineering career in test automation and SDET roles. Unit, integration, and end-to-end tests were not abstract categories. They were how we kept frequent releases from turning into frequent incidents. A recent Deep Atlas session on evaluating LLM applications pulled me back to that foundation, but with one uncomfortable change: the system under test no longer behaves the same way twice.
The answer is not to replace every assertion with an LLM judge. Build an eval pyramid instead.
This is part one of a three-part series. This post explains the pyramid and where each kind of check belongs. Part two covers repeated trials, pass@k, pass^k, and LLM judges.
Part three turns the pyramid into a runnable testing harness.
TLDR #
- Separate the system's behavior from the way you grade it. A nondeterministic agent can still have deterministic checks.
- Put cheap, fast checks at the base: schemas, exact values, unit tests, tool arguments, permissions, and final-state assertions.
- Check the result by default. Check the agent's steps only when order, permissions, safety, or cost are part of the requirement.
- Repeat important tasks because one successful run does not prove reliability.
- Add LLM judges and domain experts only for qualities that rules cannot express well, such as groundedness, completeness, and tone.
Start with the product, not the model #
A model benchmark asks, "How capable is this model in general?" A product eval asks, "Does this system do the job our users need, inside our tools, data, policies, latency budget, and failure constraints?"
Those are different questions. OpenAI makes the same distinction between frontier model evals and contextual evals built for a specific workflow. A benchmark score cannot tell you whether your support agent used the right refund policy, whether your RAG system cited the document it actually retrieved, or whether your coding agent changed the requested file and nothing else. The product boundary matters because the thing being evaluated is not just the model. It is the model, prompt, tools, retrieval, memory, orchestration, and environment working together.
Evaluate the application you are shipping, not an isolated model call. A better model may improve the result, but the product can still fail because a tool schema is confusing, the retrieved context is stale, or the test environment carries state from an earlier run.
Write the success condition in product language first. "Resolve an eligible refund without violating policy" is useful. "Score above 0.8 on helpfulness" is not, at least not yet. The metric comes after the job is clear.
The old pyramid got one thing exactly right #
Mike Cohn's original testing pyramid used unit tests at the bottom, service tests in the middle, and user-interface tests at the top. The names have shifted over time. The durable part is the shape: many small and fast tests low down, fewer broad and expensive tests near the top.
That shape matched what I saw across years of automation work. A UI test could prove that the full system worked, but it was slow, brittle, and miserable to debug. A focused unit test could not prove the whole product worked, but it told you exactly what broke in seconds. The mature approach was never to choose one. It was to keep pushing checks down until each risk sat at the cheapest layer that could catch it.
The same rule works for LLM systems.
The pyramid is a guide, not a purity test. A real eval suite will have boundaries that blur. Some end-to-end agent tasks can be graded deterministically. Some tiny outputs need a human because the domain is high stakes. The pyramid is useful because it forces a cost question: can a lower layer catch this failure with less noise?
Deterministic does not mean identical output #
There are two kinds of determinism hiding in the same conversation.
The first is generation determinism: does the system produce the same output on repeated runs? LLM applications often do not. Sampling changes tokens. Tool results change. An agent observes intermediate state and adjusts its next action. Anthropic's agent eval guide recommends running several trials because a single attempt cannot capture this variation.
The second is grading determinism: given the same completed run, does the evaluator return the same score? A unit test does. A JSON schema validator does. A database comparison does. An LLM judge might not.
Do not mix these up. A stochastic agent can take three different valid routes to the same final database state, and a deterministic state check can pass all three. That is often the ideal arrangement: let the agent be flexible, make the contract crisp.
The base of the pyramid can check things like:
- whether the output parses and matches the required schema;
- whether required fields, citations, or identifiers exist;
- whether a tool name and its arguments are valid;
- whether the agent avoided a forbidden tool or action;
- whether code compiles and passes unit tests;
- whether the final database, file system, or application state matches the goal;
- whether latency, token use, tool calls, and dollar cost stayed within limits.
These checks are fast, repeatable, and easy to debug. They also catch more than formatting. In the τ-bench agent benchmark, success is graded largely by comparing the database after a conversation with the expected goal state. The dialogue can vary. The state contract does not.
There is a catch. A deterministic check only proves what it asserts. A support agent could reach the correct database state after making an unauthorized change and then reversing it. A research agent could include all required facts while citing weak sources. A coding agent could pass tests after rewriting half the repository. The base is necessary. It is not the whole pyramid.
Check the result first. Inspect the steps when they matter #
An agent run gives you two things to inspect: the result and the steps it took to get there. Eval frameworks often call that sequence of steps a trajectory.
Checking the result is familiar. Did the agent update the right record? Does the code pass its tests? Did the answer cite the source it used?
The steps need more care because several paths can be valid. A coding agent might inspect the failing test first or read the implementation first. Both paths are fine if the resulting change is correct and focused. Comparing every run with one ideal sequence would reject useful behavior.
Sometimes the steps are part of the requirement. A refund agent may need to verify authorization before changing an account. A research agent may be forbidden from sending private data to a search tool. In those cases, reaching the right result is not enough.
Use this simpler rule: Check the result by default. Check the steps when order, permissions, safety, or cost matter.
Anthropic recommends grading outcomes instead of forcing agents through one path. LangChain's trajectory evaluators show the same trade-off with strict, unordered, subset, and superset matching.
When rules run out #
Some product requirements do not collapse into an exact string or state comparison.
Was the answer grounded in the retrieved sources? Did it include the important caveat? Was the support response firm without sounding hostile? These are real product requirements, but an exact string comparison cannot judge them.
This is where nondeterministic evaluation becomes useful. In practice, that usually means an LLM judge, a simulated user, or a human reviewer.
An LLM judge can read the task, output, and reference material, then score one quality such as correctness, groundedness, or tone. It handles reasonable variation better than exact matching and scales farther than human review.
It also brings a second model into your system, with its own mistakes and biases. The MT-Bench research found that strong model judges could match human preference judgments surprisingly well in its tests. It also found position, verbosity, and reasoning biases. A judge can prefer the first answer it sees or reward a long answer for looking more complete.
That is why people remain at the top of the pyramid. Domain experts define good examples, settle ambiguous cases, and check whether the automated scores still represent the product. When they find a repeated failure, push it down: turn a clear policy violation into a rule or add a well-tested judge criterion.
OpenAI's grader guidance recommends testing model graders against human-labeled examples before trusting them. Part two turns that idea into a practical workflow and explains how repeated trials change the reliability picture.
The pyramid is a cost model #
The useful insight in the old testing pyramid was never the exact names of the layers. It was that confidence has a cost.
At the base, deterministic assertions run cheaply and tell you exactly what broke. In the middle, repeated agent trials reveal inconsistent behavior. Higher up, LLM judges cover semantic quality but add cost and their own bias. At the top, humans decide whether the whole measurement system still matches the product and the people using it.
Push each requirement as low as it can go without losing what the requirement means. Then keep the higher layers for the failures that only appear when the full system is moving.
That is how you test a system that does not behave the same way twice. You do not demand one path. You demand a clear outcome, measure how often it arrives, and spend human judgment where it changes the decision.
Next: [how to measure repeated-run reliability with pass@k, pass^k, and calibrated LLM judges](/blog/measuring-agent-reliability). When you are ready to implement the full stack, continue to
[the hands-on build guide](/blog/build-the-eval-pyramid)and clone the
[Eval Pyramid Starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter).
Glossary #
Eval. A task, one or more trials, and grading logic used to measure an AI system.Trial. One attempt by the system on one eval task.Grader. Code, a model, or a person that scores some part of a trial.Trajectory. The full path through a trial, including messages, tool calls, intermediate results, and the final output.Outcome. The final state produced by the agent, such as changed files, a database record, or a completed booking.LLM judge. A model prompted with a rubric to grade another system's output or trajectory.
References and further reading #
Anthropic: Demystifying evals for AI agents. A practical end-to-end guide to agent tasks, trials, graders, trajectories, repeated runs, and eval design.OpenAI: How evals drive the next chapter in AI for businesses. The Specify, Measure, Improve loop and the case for contextual product evals.OpenAI API: Graders. Deterministic and model-based grader types, grader calibration, and grader hacking.. Final-state evaluation for agent tasks and theτ
-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainspass^k
reliability metric.Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Evidence for strong model judges, plus position, verbosity, and reasoning limitations.LangChain: How to evaluate your agent with trajectory evaluations. Strict, unordered, subset, superset, and model-judged trajectory evaluation.The Practical Test Pyramid. The conventional testing model this eval pyramid builds on.