{"slug": "the-eval-pyramid-testing-systems-that-do-not-behave-the-same-way-twice", "title": "The Eval Pyramid: Testing Systems That Do Not Behave the Same Way Twice", "summary": "In a new blog post, engineer and former SDET professional introduces the 'eval pyramid' for testing AI agents that behave nondeterministically, arguing that traditional testing methods remain relevant but must be adapted with new layers and a redefined concept of passing. The post, part one of a three-part series, emphasizes separating system behavior from grading, using cheap deterministic checks at the base, and reserving LLM judges for qualities like groundedness and tone. The author cites OpenAI's distinction between frontier model evals and contextual evals, and references Mike Cohn's original testing pyramid.", "body_md": "# The Eval Pyramid: Testing Systems That Do Not Behave the Same Way Twice\n\nA unit test fails the same way every time. An agent can fail five different ways, pass on the sixth attempt, and take a path nobody expected on the seventh.\n\nThat does not make traditional testing obsolete. It makes the old testing pyramid useful again, with a few new layers and a different idea of what a pass means.\n\nI spent much of the first half of my engineering career in test automation and SDET roles. Unit, integration, and end-to-end tests were not abstract categories. They were how we kept frequent releases from turning into frequent incidents. A recent Deep Atlas session on evaluating LLM applications pulled me back to that foundation, but with one uncomfortable change: the system under test no longer behaves the same way twice.\n\nThe answer is not to replace every assertion with an LLM judge. Build an **eval pyramid** instead.\n\nThis is part one of a three-part series. This post explains the pyramid and where each kind of check belongs. [Part two covers repeated trials, pass@k, pass^k, and LLM judges](/blog/measuring-agent-reliability).\n\n[Part three turns the pyramid into a runnable testing harness](/blog/build-the-eval-pyramid).\n\n## TLDR\n\n- Separate the system's behavior from the way you grade it. A nondeterministic agent can still have deterministic checks.\n- Put cheap, fast checks at the base: schemas, exact values, unit tests, tool arguments, permissions, and final-state assertions.\n- Check the result by default. Check the agent's steps only when order, permissions, safety, or cost are part of the requirement.\n- Repeat important tasks because one successful run does not prove reliability.\n- Add LLM judges and domain experts only for qualities that rules cannot express well, such as groundedness, completeness, and tone.\n\n## Start with the product, not the model\n\nA model benchmark asks, \"How capable is this model in general?\" A product eval asks, \"Does this system do the job our users need, inside our tools, data, policies, latency budget, and failure constraints?\"\n\nThose are different questions. [OpenAI makes the same distinction](https://openai.com/index/evals-drive-next-chapter-of-ai/) between frontier model evals and contextual evals built for a specific workflow. A benchmark score cannot tell you whether your support agent used the right refund policy, whether your RAG system cited the document it actually retrieved, or whether your coding agent changed the requested file and nothing else. The product boundary matters because the thing being evaluated is not just the model. It is the model, prompt, tools, retrieval, memory, orchestration, and environment working together.\n\nEvaluate the application you are shipping, not an isolated model call. A better model may improve the result, but the product can still fail because a tool schema is confusing, the retrieved context is stale, or the test environment carries state from an earlier run.\n\nWrite the success condition in product language first. \"Resolve an eligible refund without violating policy\" is useful. \"Score above 0.8 on helpfulness\" is not, at least not yet. The metric comes after the job is clear.\n\n## The old pyramid got one thing exactly right\n\n[Mike Cohn's original testing pyramid](https://martinfowler.com/articles/practical-test-pyramid.html#TheTestPyramid) used unit tests at the bottom, service tests in the middle, and user-interface tests at the top. The names have shifted over time. The durable part is the shape: many small and fast tests low down, fewer broad and expensive tests near the top.\n\nThat shape matched what I saw across years of automation work. A UI test could prove that the full system worked, but it was slow, brittle, and miserable to debug. A focused unit test could not prove the whole product worked, but it told you exactly what broke in seconds. The mature approach was never to choose one. It was to keep pushing checks down until each risk sat at the cheapest layer that could catch it.\n\nThe same rule works for LLM systems.\n\nThe pyramid is a guide, not a purity test. A real eval suite will have boundaries that blur. Some end-to-end agent tasks can be graded deterministically. Some tiny outputs need a human because the domain is high stakes. The pyramid is useful because it forces a cost question: can a lower layer catch this failure with less noise?\n\n## Deterministic does not mean identical output\n\nThere are two kinds of determinism hiding in the same conversation.\n\nThe first is **generation determinism**: does the system produce the same output on repeated runs? LLM applications often do not. Sampling changes tokens. Tool results change. An agent observes intermediate state and adjusts its next action. [Anthropic's agent eval guide](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) recommends running several trials because a single attempt cannot capture this variation.\n\nThe second is **grading determinism**: given the same completed run, does the evaluator return the same score? A unit test does. A JSON schema validator does. A database comparison does. An LLM judge might not.\n\nDo not mix these up. A stochastic agent can take three different valid routes to the same final database state, and a deterministic state check can pass all three. That is often the ideal arrangement: let the agent be flexible, make the contract crisp.\n\nThe base of the pyramid can check things like:\n\n- whether the output parses and matches the required schema;\n- whether required fields, citations, or identifiers exist;\n- whether a tool name and its arguments are valid;\n- whether the agent avoided a forbidden tool or action;\n- whether code compiles and passes unit tests;\n- whether the final database, file system, or application state matches the goal;\n- whether latency, token use, tool calls, and dollar cost stayed within limits.\n\nThese checks are fast, repeatable, and easy to debug. They also catch more than formatting. In the [ τ-bench agent benchmark](https://arxiv.org/abs/2406.12045), success is graded largely by comparing the database after a conversation with the expected goal state. The dialogue can vary. The state contract does not.\n\nThere is a catch. A deterministic check only proves what it asserts. A support agent could reach the correct database state after making an unauthorized change and then reversing it. A research agent could include all required facts while citing weak sources. A coding agent could pass tests after rewriting half the repository. The base is necessary. It is not the whole pyramid.\n\n## Check the result first. Inspect the steps when they matter\n\nAn agent run gives you two things to inspect: the result and the steps it took to get there. Eval frameworks often call that sequence of steps a **trajectory**.\n\nChecking the result is familiar. Did the agent update the right record? Does the code pass its tests? Did the answer cite the source it used?\n\nThe steps need more care because several paths can be valid. A coding agent might inspect the failing test first or read the implementation first. Both paths are fine if the resulting change is correct and focused. Comparing every run with one ideal sequence would reject useful behavior.\n\nSometimes the steps are part of the requirement. A refund agent may need to verify authorization before changing an account. A research agent may be forbidden from sending private data to a search tool. In those cases, reaching the right result is not enough.\n\nUse this simpler rule:\n\n**Check the result by default. Check the steps when order, permissions, safety, or cost matter.**\n\n[Anthropic recommends grading outcomes](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) instead of forcing agents through one path. [LangChain's trajectory evaluators](https://docs.langchain.com/langsmith/trajectory-evals) show the same trade-off with strict, unordered, subset, and superset matching.\n\n## When rules run out\n\nSome product requirements do not collapse into an exact string or state comparison.\n\nWas the answer grounded in the retrieved sources? Did it include the important caveat? Was the support response firm without sounding hostile? These are real product requirements, but an exact string comparison cannot judge them.\n\nThis is where nondeterministic evaluation becomes useful. In practice, that usually means an LLM judge, a simulated user, or a human reviewer.\n\nAn LLM judge can read the task, output, and reference material, then score one quality such as correctness, groundedness, or tone. It handles reasonable variation better than exact matching and scales farther than human review.\n\nIt also brings a second model into your system, with its own mistakes and biases. The [MT-Bench research](https://arxiv.org/abs/2306.05685) found that strong model judges could match human preference judgments surprisingly well in its tests. It also found position, verbosity, and reasoning biases. A judge can prefer the first answer it sees or reward a long answer for looking more complete.\n\nThat is why people remain at the top of the pyramid. Domain experts define good examples, settle ambiguous cases, and check whether the automated scores still represent the product. When they find a repeated failure, push it down: turn a clear policy violation into a rule or add a well-tested judge criterion.\n\n[OpenAI's grader guidance](https://developers.openai.com/api/docs/guides/graders) recommends testing model graders against human-labeled examples before trusting them. Part two turns that idea into a practical workflow and explains how repeated trials change the reliability picture.\n\n## The pyramid is a cost model\n\nThe useful insight in the old testing pyramid was never the exact names of the layers. It was that confidence has a cost.\n\nAt the base, deterministic assertions run cheaply and tell you exactly what broke. In the middle, repeated agent trials reveal inconsistent behavior. Higher up, LLM judges cover semantic quality but add cost and their own bias. At the top, humans decide whether the whole measurement system still matches the product and the people using it.\n\nPush each requirement as low as it can go without losing what the requirement means. Then keep the higher layers for the failures that only appear when the full system is moving.\n\nThat is how you test a system that does not behave the same way twice. You do not demand one path. You demand a clear outcome, measure how often it arrives, and spend human judgment where it changes the decision.\n\nNext: [how to measure repeated-run reliability with pass@k, pass^k, and calibrated LLM judges](/blog/measuring-agent-reliability). When you are ready to implement the full stack, continue to\n\n[the hands-on build guide](/blog/build-the-eval-pyramid)and clone the\n\n[Eval Pyramid Starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter).\n\n## Glossary\n\n**Eval.** A task, one or more trials, and grading logic used to measure an AI system.**Trial.** One attempt by the system on one eval task.**Grader.** Code, a model, or a person that scores some part of a trial.**Trajectory.** The full path through a trial, including messages, tool calls, intermediate results, and the final output.**Outcome.** The final state produced by the agent, such as changed files, a database record, or a completed booking.**LLM judge.** A model prompted with a rubric to grade another system's output or trajectory.\n\n## References and further reading\n\n[Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). A practical end-to-end guide to agent tasks, trials, graders, trajectories, repeated runs, and eval design.[OpenAI: How evals drive the next chapter in AI for businesses](https://openai.com/index/evals-drive-next-chapter-of-ai/). The Specify, Measure, Improve loop and the case for contextual product evals.[OpenAI API: Graders](https://developers.openai.com/api/docs/guides/graders). Deterministic and model-based grader types, grader calibration, and grader hacking.. Final-state evaluation for agent tasks and the`τ`\n\n-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains`pass^k`\n\nreliability metric.[Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685). Evidence for strong model judges, plus position, verbosity, and reasoning limitations.[LangChain: How to evaluate your agent with trajectory evaluations](https://docs.langchain.com/langsmith/trajectory-evals). Strict, unordered, subset, superset, and model-judged trajectory evaluation.[The Practical Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). The conventional testing model this eval pyramid builds on.", "url": "https://wpnews.pro/news/the-eval-pyramid-testing-systems-that-do-not-behave-the-same-way-twice", "canonical_source": "https://bharad.dev/blog/eval-pyramid", "published_at": "2026-08-09 00:00:00+00:00", "updated_at": "2026-08-10 07:20:03.597426+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools", "ai-ethics"], "entities": ["OpenAI", "Mike Cohn", "Deep Atlas"], "alternates": {"html": "https://wpnews.pro/news/the-eval-pyramid-testing-systems-that-do-not-behave-the-same-way-twice", "markdown": "https://wpnews.pro/news/the-eval-pyramid-testing-systems-that-do-not-behave-the-same-way-twice.md", "text": "https://wpnews.pro/news/the-eval-pyramid-testing-systems-that-do-not-behave-the-same-way-twice.txt", "jsonld": "https://wpnews.pro/news/the-eval-pyramid-testing-systems-that-do-not-behave-the-same-way-twice.jsonld"}}