Designing the Evaluation Framework A tutorial on building an agentic AI evaluation framework lays out four evaluation layers — component, step, trajectory, and outcome — and five core abstractions — Case, Run, Grader, Score, and Report — in roughly 200 lines of code. The lesson warns that relying only on the outcome layer leaves teams unable to explain a score drop, citing an example where an outcome score fell from 88% to 81% with no visible cause, while step and trajectory layers can trace the regression to tool-selection accuracy after a get_invoices call. The framework is presented as the prerequisite for the course's later lessons on eval-driven agent architecture. · Agentic AI · 8 min read 📋 Prerequisites - Eval-Driven Agent Architecture previous lesson 🎯 What You'll Learn - Describe the four evaluation layers and what each one catches - Implement the five core abstractions of an evaluation framework: Case, Run, Grader, Score, and Report - Combine multiple graders into a weighted, gated scorecard - Run an evaluation harness concurrently and produce a comparable report Why a Framework, Not a Script Your first eval is usually a script: loop over some inputs, run the agent, check something, print a percentage. That’s the right way to start. It stops scaling as soon as you want to: - Add a second kind of check without rewriting the loop - Compare two agent versions on the same cases - Run each case several times and report variance - Find out which check failed on which case, and open its trace An evaluation framework is the small set of abstractions that make all of that routine. It doesn’t need to be big. The version we build in this lesson is about 200 lines, and it’s the backbone of everything that follows. The Four Evaluation Layers Agents fail at different granularities. A good framework evaluates at all four, because each layer catches failures the others miss. ┌────────────────────────────────────────────────────────────┐ │ 4. OUTCOME Did the task succeed end to end? │ few, expensive, decisive ├────────────────────────────────────────────────────────────┤ │ 3. TRAJECTORY Was the path acceptable, safe, efficient? │ ├────────────────────────────────────────────────────────────┤ │ 2. STEP Was each individual decision correct? │ ├────────────────────────────────────────────────────────────┤ │ 1. COMPONENT Does each piece work in isolation? │ many, cheap, diagnostic └────────────────────────────────────────────────────────────┘ Layer 1: Component Test pieces in isolation: a retrieval function returns the right KB article; a tool’s argument parser handles malformed dates; the intent classifier routes “I was charged twice” to billing. These are close to ordinary unit tests. They’re fast and they localize failures precisely. Layer 2: Step Freeze the conversation at a specific point and ask: given this exact context, does the model make the right next decision? For Quill: given a message history where get invoices just returned two identical charges, does the next call refund the duplicate rather than the original? Step evals are the agent equivalent of unit tests for decisions. They’re cheap one model call each , deterministic in their setup, and the best tool for regression-testing a specific bug fix. step case = { "messages": ... , history up to the decision point "expect tool": "issue refund", "expect args": {"invoice id": "INV-2291"}, } Layer 3: Trajectory Run the full loop and grade the path: required tools called, forbidden tools avoided, correct ordering, no redundant calls, within a step budget. Covered in depth in Lesson 7. Layer 4: Outcome Run the full loop and grade the end state and response. Did the right refund happen? Was the customer told accurately? This is the number stakeholders care about. The common mistake is having only Layer 4. Your outcome score drops from 88% to 81%, and you have no idea why. With step and trajectory layers, you can see that tool-selection accuracy after get invoices fell, which points directly at the prompt change that caused it. The Five Core Abstractions Case ──▶ Runner ──▶ Run ──▶ Grader s ──▶ Score s ──▶ Report Case One test scenario: input, setup state, expectations, and metadata for slicing results. python from dataclasses import dataclass, field from typing import Any @dataclass frozen=True class Case: id: str input: str initial state: dict expected: dict = field default factory=dict free-form, read by graders tags: tuple str, ... = e.g. "intent:duplicate", "risk:high" split: str = "regression" "smoke" | "regression" | "adversarial" expected is deliberately loose. Different graders read different keys: expected "refund" for an outcome grader, expected "forbidden tools" for a trajectory grader, expected "rubric notes" for a judge. Run The result of executing one case once. This is the RunResult from Lesson 2 plus identifying metadata. @dataclass class Run: case id: str trial: int config fingerprint: str response: str trace: "Trace" final state: dict stopped reason: str latency ms: float cost usd: float Grader A function from Case, Run to a Score . That’s the entire contract, and keeping it that narrow is what lets you mix deterministic checks, trajectory scorers, and LLM judges freely. python from typing import Protocol class Grader Protocol : name: str def grade self, case: Case, run: Run - "Score": ... Score A grader’s verdict. Always include a reason. A score without a reason is unreadable when you’re debugging 30 failures at midnight. @dataclass class Score: grader: str value: float 0.0 to 1.0 passed: bool reason: str = "" details: dict = field default factory=dict Normalize everything to 0–1 with a passed boolean. A judge’s 1–5 rating becomes rating - 1 / 4 with a pass threshold; a binary assertion is 0 or 1. Report The aggregation over many runs: overall and per-grader pass rates, slices by tag, per-case details, cost, latency, and comparison against a baseline. Lesson 10 adds confidence intervals; Lesson 12 adds regression gates. Grader Composition: Gates and Weights Not all graders are equal. Some failures are disqualifying regardless of everything else. For Quill, refunding over the policy limit is a failure even if the customer is delighted. Model this with two kinds of graders: - Gates : must pass, or the run fails outright - Weighted : contribute to a quality score @dataclass class Scorecard: gates: list Grader weighted: list tuple Grader, float pass threshold: float = 0.7 def evaluate self, case: Case, run: Run - dict: gate scores = g.grade case, run for g in self.gates weighted scores = g.grade case, run , w for g, w in self.weighted gates ok = all s.passed for s in gate scores total w = sum w for , w in weighted scores or 1.0 quality = sum s.value w for s, w in weighted scores / total w return { "passed": gates ok and quality = self.pass threshold, "quality": quality if gates ok else 0.0, "scores": gate scores + s for s, in weighted scores , } A practical Quill scorecard: | Grader | Type | Weight | |---|---|---| | No refund over policy limit | Gate | — | | No action on wrong customer | Gate | — | | Completed didn’t hit max steps | Gate | — | | Correct final state | Weighted | 0.5 | | Trajectory acceptable | Weighted | 0.2 | | Response quality judge | Weighted | 0.3 | The Runner The runner executes cases against an agent factory, handling concurrency, repeated trials, timeouts, and errors. Errors are results, not crashes: an agent that throws is a failed run that should appear in the report. python import asyncio, time class Runner: def init self, agent factory, config, trials: int = 1, concurrency: int = 8, timeout s: float = 120 : self.agent factory, self.config = agent factory, config self.trials, self.timeout s = trials, timeout s self.sem = asyncio.Semaphore concurrency async def run one self, case: Case, trial: int - Run: async with self.sem: agent, state = self.agent factory self.config, case.initial state t0 = time.perf counter try: result = await asyncio.wait for asyncio.to thread agent.run, case.input, state , self.timeout s stopped, response = result.stopped reason, result.response trace, final state = result.trace, result.final state except Exception as e: stopped, response = f"error:{type e . name }", "" trace, final state = agent.tracer.trace, state.snapshot latency = time.perf counter - t0 1000 return Run case.id, trial, self.config.fingerprint , response, trace, final state, stopped, latency, estimate cost trace, self.config.model async def run self, cases: list Case - list Run : jobs = self. run one c, t for c in cases for t in range self.trials return await asyncio.gather jobs Two details matter: - Fresh state per run. Each run gets its own copy of initial state . Shared mutable state between concurrent runs is the most common source of flaky eval results. - Rate limits. Concurrency is bounded by a semaphore. Provider rate limits will otherwise turn into a wave of errors that look like agent failures. The Report python from collections import defaultdict def build report cases: list Case , runs: list Run , scorecard: Scorecard - dict: by id = {c.id: c for c in cases} rows = for run in runs: case = by id run.case id result = scorecard.evaluate case, run rows.append {"case": case, "run": run, result} def rate rs : return sum r "passed" for r in rs / len rs if rs else 0.0 by tag = defaultdict list for r in rows: for t in r "case" .tags: by tag t .append r by grader = defaultdict list for r in rows: for s in r "scores" : by grader s.grader .append s.passed return { "config": runs 0 .config fingerprint, "n runs": len rows , "pass rate": rate rows , "by tag": {t: rate rs for t, rs in sorted by tag.items }, "by grader": {g: sum v / len v for g, v in by grader.items }, "mean cost usd": sum r "run" .cost usd for r in rows / len rows , "p50 latency ms": sorted r "run" .latency ms for r in rows len rows // 2 , "failures": {"case": r "case" .id, "trial": r "run" .trial, "reasons": s.reason for s in r "scores" if not s.passed , "trace id": r "run" .trace.trace id} for r in rows if not r "passed" , } The failures list is the most-read part of any report. Each entry links to a trace ID, so “why did case 47 fail?” is a single click, not a reproduction effort. Build vs Buy You don’t have to build all of this yourself. Here’s how hosted and open-source tools map onto the abstractions: | Abstraction | Examples | |---|---| | Case / dataset | LangSmith datasets, Braintrust datasets, Langfuse datasets, plain JSONL in git | | Runner | Inspect UK AISI , OpenAI Evals, promptfoo, Braintrust Eval , pytest | | Grader | DeepEval, Ragas, autoevals, your own functions | | Report | Braintrust, LangSmith experiments, Langfuse, a Markdown file posted to the PR | The recommendation in this course: own the Case, Grader, and Scorecard definitions in your repo , because they encode your product’s definition of “correct.” Rent the runner, storage, and UI if it saves you time. If you swap vendors later, your test cases and graders shouldn’t have to change. Key Takeaways - Evaluate at four layers: component , step , trajectory , outcome . Outcome-only evals can’t explain regressions. - Five abstractions cover almost everything: Case , Run , Grader , Score , Report - Keep the grader contract narrow: Case, Run → Score , always with a reason - Separate gates disqualifying failures from weighted quality scores - Errors are results; give each run fresh state; bound concurrency - Own the definitions of correctness in your repo; rent infrastructure if it helps Exercise Implement Case , Run , Score , Scorecard , Runner , and build report for your agent. Write three trivial graders to prove the plumbing works: 1. CompletedGrader : passes if stopped reason == "final answer" 2. MaxStepsGrader : passes if the trace has at most 8 tool spans 3. NonEmptyResponseGrader : passes if the response has at least 20 characters Run five cases with three trials each and print the report. Next lesson, you’ll replace those five cases with a real dataset.