{"slug": "why-agent-evaluation-is-different", "title": "Why Agent Evaluation Is Different", "summary": "Agent evaluation requires statistical, multi-run testing because agents are loops rather than functions, according to a tutorial on agentic AI evaluation that outlines five properties making agents harder to evaluate than models. The tutorial notes that non-determinism at every step means one run per test case is not a measurement, that many valid trajectories exist (illustrated by a billing-support agent named Quill that can resolve a duplicate charge via refund, knowledge-base lookup, or human escalation), and that side effects such as a $180 erroneous refund require evaluating state changes in sandboxed or mocked environments. The piece also describes a \"vibe check ceiling\" that teams hit when multiple people change an agent, task types outgrow one person's memory, a model upgrade ships without a safety verdict, or a customer failure can't be traced as new or pre-existing.", "body_md": "· Agentic AI · 7 min read\n\n### 📋 Prerequisites\n\n- Experience building at least one tool-using agent\n\n### 🎯 What You'll Learn\n\n- Explain the five properties that make agents harder to evaluate than models\n- Calculate how per-step reliability compounds across a multi-step trajectory\n- Distinguish outcome, trajectory, and process quality, and know when each matters\n- Recognize the common anti-patterns teams fall into before they have an eval framework\n\n## The Vibe Check Ceiling\n\nHere’s how most agents get evaluated in their first month. An engineer writes ten queries they think are representative, runs the agent, reads the outputs, and decides it “looks good.” When they change the prompt, they rerun some of those queries, read again, and decide it “looks better.”\n\nThis works surprisingly well at first. It stops working at a predictable point. Call it the **vibe check ceiling**, and you hit it when one of these becomes true:\n\n- More than one person changes the agent, and they disagree about whether a change helped\n- The agent handles more distinct task types than one person can hold in their head\n- A model upgrade ships and nobody can say whether it’s safe to adopt\n- A customer reports a failure, and you can’t tell if it’s new or has always been there\n\nPast this ceiling, every change is a coin flip you can’t see the result of. An evaluation framework is how you get past it. Before you build one, you need to understand why the tools you already know from ML evaluation aren’t enough.\n\n## Model Evaluation, Briefly\n\nClassic model evaluation has a clean shape:\n\n1. A fixed input\n2. A single output\n3. A label to compare against\n4. A metric (accuracy, F1, BLEU, ROUGE) that aggregates over a test set\n\nEven LLM benchmarks mostly keep this shape: a question, an answer, a reference, a score. The model is a function. You evaluate the function.\n\nAn agent isn’t a function. It’s a **loop**. It reads a goal, decides on an action, observes the result, and decides again, often a dozen times, while changing the state of the world as it goes. Every property that made model evaluation tractable breaks somewhere in that loop.\n\n## Five Properties That Make Agents Hard to Evaluate\n\n### 1. Non-Determinism at Every Step\n\nRun the same model call twice at temperature 0 and you’ll often get the same answer. Run the same *agent* twice and small differences in one step change which tool gets called next, which changes the observation, which changes everything after. A single run tells you what *can* happen, not what *usually* happens.\n\nThis means **one run per test case isn’t a measurement**. Lesson 10 covers how many runs you need and how to report the variance. For now, accept that agent evals are inherently statistical.\n\n### 2. Many Valid Paths\n\nAsk Quill (our billing-support agent) “Why was I charged twice in March?” One good trajectory:\n\n```\nlookup_customer → get_invoices → (finds duplicate) → issue_refund → respond\n```\n\nAnother equally good one:\n\n```\nlookup_customer → get_invoices → search_kb(\"duplicate charge policy\") → issue_refund → respond\n```\n\nA third that’s arguably better for a high-value account:\n\n```\nlookup_customer → get_invoices → escalate_to_human → respond\n```\n\nThere’s no single “label” trajectory. Exact-match comparison against one reference path punishes correct behavior. Lesson 7 shows how to grade trajectories against *constraints* (must call X before Y, must never call Z) instead of exact sequences.\n\n### 3. Side Effects\n\nWhen a classifier is wrong, you get a wrong label. When Quill is wrong, a customer gets $180 they weren’t owed, or a legitimate refund doesn’t happen. Agents **act**, and the actions are often the thing you most need to evaluate.\n\nTwo consequences follow:\n\n- You need to evaluate **state changes** , not just text. Did the refund record get created, for the right invoice, for the right amount?\n- You need **sandboxed or mocked environments** so evaluation doesn’t refund real money. Lesson 2 covers how to architect for this.\n\n### 4. Errors Compound\n\nThis is the property that surprises people most, so it gets its own section below.\n\n### 5. The Final Answer Hides the Process\n\nAn agent can produce the right final answer by luck: it guessed the invoice instead of looking it up, and the guess happened to be right. It can also produce a reasonable-sounding wrong answer after a perfectly good process, because one tool returned stale data. If you only grade the final answer, you can’t tell these apart, and you can’t fix what you can’t see.\n\n## The Compounding-Error Math\n\nSuppose each step of an agent’s trajectory succeeds independently with probability *p*. A task that needs *n* steps succeeds with probability *p* to the power *n*.\n\n```\nfor p in [0.99, 0.95, 0.90]:\n    for n in [5, 10, 20]:\n        print(f\"per-step {p:.0%}, {n:>2} steps → task success {p**n:.1%}\")\nper-step 99%,  5 steps → task success 95.1%\nper-step 99%, 10 steps → task success 90.4%\nper-step 99%, 20 steps → task success 81.8%\nper-step 95%,  5 steps → task success 77.4%\nper-step 95%, 10 steps → task success 59.9%\nper-step 95%, 20 steps → task success 35.8%\nper-step 90%,  5 steps → task success 59.0%\nper-step 90%, 10 steps → task success 34.9%\nper-step 90%, 20 steps → task success 12.2%\n```\n\nAn agent that’s “95% accurate” at each decision fails **four out of ten** ten-step tasks. Real agents aren’t quite this bad, because they can notice errors and recover, which breaks the independence assumption. But that recovery is itself a behavior you have to measure.\n\nThe practical lesson has two parts:\n\n1. **Step-level metrics are necessary but not sufficient.** A 2% improvement in tool-selection accuracy can be a 15% improvement in task success on long tasks. You need both numbers.\n2. **Long tasks need disproportionate eval attention.** They’re where compounding bites, and they’re usually underrepresented in hand-written test sets because they’re tedious to write.\n\n## Three Things You Can Evaluate\n\nEvery agent eval measures some mix of three things. Being explicit about which one you’re measuring avoids most confused eval designs.\n\n| Dimension | Question | Example for Quill | Typical grader | \n|---|---|---|---|\n| **Outcome** | Did the world end up in the right state? | A refund for exactly $49 exists on invoice INV-2291 | State assertion | \n| **Trajectory** | Did the agent take an acceptable path? | Looked up the customer before refunding; never refunded twice | Trajectory constraints | \n| **Response** | Was the communication good? | Clear, accurate, polite, no invented policy | Rubric / LLM judge | \n\nOutcome is usually the most important and the easiest to grade reliably, if your architecture exposes the state. Trajectory catches lucky successes and wasteful paths. Response quality is where LLM judges earn their keep, and also where they’re most dangerous if uncalibrated.\n\n## Anti-Patterns Before a Framework Exists\n\nYou’ll recognize most of these. Each one is a symptom the rest of the course fixes.\n\n**The demo set.** Ten hand-picked queries that the agent was tuned on. They measure memorization of the demo, not capability. *Fixed in Lesson 4.*\n\n**Final-answer string matching.** `assert \"refund\" in response`. Passes when the agent says “I can’t issue a refund.” *Fixed in Lesson 5.*\n\n**The unvalidated judge.** “We use GPT-4 as a judge, it gives us 8.2/10.” Nobody has checked whether the judge agrees with a human. *Fixed in Lesson 6.*\n\n**Single-run scoring.** A change goes from 82% to 85% on one run of 40 cases. That’s well inside the noise. *Fixed in Lesson 10.*\n\n**Eval in a notebook.** The eval exists, but someone has to remember to run it, so it runs before big launches and never in between. *Fixed in Lesson 12.*\n\n**Offline-only.** The test set was written once, in January. Production traffic in November looks nothing like it. *Fixed in Lesson 11.*\n\n## What a Mature Eval Practice Looks Like\n\nBy the end of this course, the target looks like this:\n\n- A **versioned golden dataset** of a few hundred cases, stratified by intent and risk, that grows every week from production failures\n- A **grader stack** where cheap deterministic checks carry most of the weight, and a calibrated judge handles what’s left\n- **Every run traced** , so a failing case can be opened and inspected step by step\n- **Reliability reported honestly** , with multiple trials, confidence intervals, and cost and latency alongside quality\n- A **CI gate** that runs a fast suite on every pull request and a full suite nightly\n- **Online evaluators** watching sampled production traffic, feeding new cases back into the dataset\n\nNone of that is exotic. It’s ordinary engineering discipline applied to a system that happens to be non-deterministic. The hardest part is usually the first step: designing the agent so it can be observed at all. That’s the next lesson.\n\n## Key Takeaways\n\n- An agent is a loop with side effects, not a function, so model-evaluation habits don’t transfer\n- Non-determinism means a single run isn’t a measurement; agent evals are statistical\n- Many trajectories can be valid, so grade against constraints, not one reference path\n- Per-step errors compound: 95% per step is about 60% task success over ten steps\n- Separate **outcome** ,**trajectory** , and**response** quality, and pick graders for each deliberately\n\n## Exercise\n\nTake an agent you’ve built (or use Quill’s tool list from the course overview). Write down:\n\n1. The three most common task types it handles, and the typical number of steps for each\n2. For each, what the correct *outcome* is in terms of state, not text\n3. One trajectory that reaches the right outcome but that you’d still consider a failure, and why\n\nKeep this list. You’ll turn it into the seed of a golden dataset in Lesson 4.", "url": "https://wpnews.pro/news/why-agent-evaluation-is-different", "canonical_source": "https://superml.org/tutorials/agent-eval-why-different", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:49.705528+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Quill"], "also_reported_by": [], "takeaway": "Because a single agent run only shows what can happen rather than what usually happens, evaluate each test case across multiple runs and grade trajectories against constraints (must call X before Y, must never call Z) instead of exact-matching one reference path.", "recommendation": {"advice": "Run each agent test case multiple times and report variance, and grade trajectories against ordering constraints rather than exact sequences, since valid paths differ and one run is not a measurement.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/why-agent-evaluation-is-different", "markdown": "https://wpnews.pro/news/why-agent-evaluation-is-different.md", "text": "https://wpnews.pro/news/why-agent-evaluation-is-different.txt", "jsonld": "https://wpnews.pro/news/why-agent-evaluation-is-different.jsonld"}}