{"slug": "designing-the-evaluation-framework", "title": "Designing the Evaluation Framework", "summary": "A tutorial on building an agentic AI evaluation framework lays out four evaluation layers — component, step, trajectory, and outcome — and five core abstractions — Case, Run, Grader, Score, and Report — in roughly 200 lines of code. The lesson warns that relying only on the outcome layer leaves teams unable to explain a score drop, citing an example where an outcome score fell from 88% to 81% with no visible cause, while step and trajectory layers can trace the regression to tool-selection accuracy after a get_invoices call. The framework is presented as the prerequisite for the course's later lessons on eval-driven agent architecture.", "body_md": "· Agentic AI · 8 min read\n\n### 📋 Prerequisites\n\n- Eval-Driven Agent Architecture (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Describe the four evaluation layers and what each one catches\n- Implement the five core abstractions of an evaluation framework: Case, Run, Grader, Score, and Report\n- Combine multiple graders into a weighted, gated scorecard\n- Run an evaluation harness concurrently and produce a comparable report\n\n## Why a Framework, Not a Script\n\nYour first eval is usually a script: loop over some inputs, run the agent, check something, print a percentage. That’s the right way to start. It stops scaling as soon as you want to:\n\n- Add a second kind of check without rewriting the loop\n- Compare two agent versions on the same cases\n- Run each case several times and report variance\n- Find out *which* check failed on*which* case, and open its trace\n\nAn evaluation framework is the small set of abstractions that make all of that routine. It doesn’t need to be big. The version we build in this lesson is about 200 lines, and it’s the backbone of everything that follows.\n\n## The Four Evaluation Layers\n\nAgents fail at different granularities. A good framework evaluates at all four, because each layer catches failures the others miss.\n\n```\n ┌────────────────────────────────────────────────────────────┐\n │ 4. OUTCOME     Did the task succeed end to end?            │  few, expensive, decisive\n ├────────────────────────────────────────────────────────────┤\n │ 3. TRAJECTORY  Was the path acceptable, safe, efficient?   │\n ├────────────────────────────────────────────────────────────┤\n │ 2. STEP        Was each individual decision correct?       │\n ├────────────────────────────────────────────────────────────┤\n │ 1. COMPONENT   Does each piece work in isolation?          │  many, cheap, diagnostic\n └────────────────────────────────────────────────────────────┘\n```\n\n### Layer 1: Component\n\nTest pieces in isolation: a retrieval function returns the right KB article; a tool’s argument parser handles malformed dates; the intent classifier routes “I was charged twice” to billing. These are close to ordinary unit tests. They’re fast and they localize failures precisely.\n\n### Layer 2: Step\n\nFreeze the conversation at a specific point and ask: *given this exact context, does the model make the right next decision?* For Quill: given a message history where `get_invoices` just returned two identical charges, does the next call refund the duplicate rather than the original?\n\nStep evals are the agent equivalent of unit tests for decisions. They’re cheap (one model call each), deterministic in their setup, and the best tool for regression-testing a specific bug fix.\n\n```\nstep_case = {\n    \"messages\": [...],               # history up to the decision point\n    \"expect_tool\": \"issue_refund\",\n    \"expect_args\": {\"invoice_id\": \"INV-2291\"},\n}\n```\n\n### Layer 3: Trajectory\n\nRun the full loop and grade the path: required tools called, forbidden tools avoided, correct ordering, no redundant calls, within a step budget. Covered in depth in Lesson 7.\n\n### Layer 4: Outcome\n\nRun the full loop and grade the end state and response. Did the right refund happen? Was the customer told accurately? This is the number stakeholders care about.\n\n**The common mistake is having only Layer 4.** Your outcome score drops from 88% to 81%, and you have no idea why. With step and trajectory layers, you can see that tool-selection accuracy after `get_invoices` fell, which points directly at the prompt change that caused it.\n\n## The Five Core Abstractions\n\n```\n Case ──▶ Runner ──▶ Run ──▶ Grader(s) ──▶ Score(s) ──▶ Report\n```\n\n### Case\n\nOne test scenario: input, setup state, expectations, and metadata for slicing results.\n\n``` python\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\n@dataclass(frozen=True)\nclass Case:\n    id: str\n    input: str\n    initial_state: dict\n    expected: dict = field(default_factory=dict)  # free-form, read by graders\n    tags: tuple[str, ...] = ()                    # e.g. (\"intent:duplicate\", \"risk:high\")\n    split: str = \"regression\"                     # \"smoke\" | \"regression\" | \"adversarial\"\n```\n\n`expected` is deliberately loose. Different graders read different keys: `expected[\"refund\"]` for an outcome grader, `expected[\"forbidden_tools\"]` for a trajectory grader, `expected[\"rubric_notes\"]` for a judge.\n\n### Run\n\nThe result of executing one case once. This is the `RunResult` from Lesson 2 plus identifying metadata.\n\n```\n@dataclass\nclass Run:\n    case_id: str\n    trial: int\n    config_fingerprint: str\n    response: str\n    trace: \"Trace\"\n    final_state: dict\n    stopped_reason: str\n    latency_ms: float\n    cost_usd: float\n```\n\n### Grader\n\nA function from `(Case, Run)` to a `Score`. That’s the entire contract, and keeping it that narrow is what lets you mix deterministic checks, trajectory scorers, and LLM judges freely.\n\n``` python\nfrom typing import Protocol\n\nclass Grader(Protocol):\n    name: str\n    def grade(self, case: Case, run: Run) -> \"Score\": ...\n```\n\n### Score\n\nA grader’s verdict. Always include a reason. A score without a reason is unreadable when you’re debugging 30 failures at midnight.\n\n```\n@dataclass\nclass Score:\n    grader: str\n    value: float            # 0.0 to 1.0\n    passed: bool\n    reason: str = \"\"\n    details: dict = field(default_factory=dict)\n```\n\nNormalize everything to 0–1 with a `passed` boolean. A judge’s 1–5 rating becomes `(rating - 1) / 4` with a pass threshold; a binary assertion is 0 or 1.\n\n### Report\n\nThe aggregation over many runs: overall and per-grader pass rates, slices by tag, per-case details, cost, latency, and comparison against a baseline. Lesson 10 adds confidence intervals; Lesson 12 adds regression gates.\n\n## Grader Composition: Gates and Weights\n\nNot all graders are equal. Some failures are disqualifying regardless of everything else. For Quill, refunding over the policy limit is a failure even if the customer is delighted. Model this with two kinds of graders:\n\n- **Gates** : must pass, or the run fails outright\n- **Weighted** : contribute to a quality score\n\n```\n@dataclass\nclass Scorecard:\n    gates: list[Grader]\n    weighted: list[tuple[Grader, float]]\n    pass_threshold: float = 0.7\n\n    def evaluate(self, case: Case, run: Run) -> dict:\n        gate_scores = [g.grade(case, run) for g in self.gates]\n        weighted_scores = [(g.grade(case, run), w) for g, w in self.weighted]\n\n        gates_ok = all(s.passed for s in gate_scores)\n        total_w = sum(w for _, w in weighted_scores) or 1.0\n        quality = sum(s.value * w for s, w in weighted_scores) / total_w\n\n        return {\n            \"passed\": gates_ok and quality >= self.pass_threshold,\n            \"quality\": quality if gates_ok else 0.0,\n            \"scores\": gate_scores + [s for s, _ in weighted_scores],\n        }\n```\n\nA practical Quill scorecard:\n\n| Grader | Type | Weight | \n|---|---|---|\n| No refund over policy limit | Gate | — | \n| No action on wrong customer | Gate | — | \n| Completed (didn’t hit max steps) | Gate | — | \n| Correct final state | Weighted | 0.5 | \n| Trajectory acceptable | Weighted | 0.2 | \n| Response quality (judge) | Weighted | 0.3 | \n\n## The Runner\n\nThe runner executes cases against an agent factory, handling concurrency, repeated trials, timeouts, and errors. Errors are results, not crashes: an agent that throws is a failed run that should appear in the report.\n\n``` python\nimport asyncio, time\n\nclass Runner:\n    def __init__(self, agent_factory, config, trials: int = 1, concurrency: int = 8,\n                 timeout_s: float = 120):\n        self.agent_factory, self.config = agent_factory, config\n        self.trials, self.timeout_s = trials, timeout_s\n        self.sem = asyncio.Semaphore(concurrency)\n\n    async def _run_one(self, case: Case, trial: int) -> Run:\n        async with self.sem:\n            agent, state = self.agent_factory(self.config, case.initial_state)\n            t0 = time.perf_counter()\n            try:\n                result = await asyncio.wait_for(\n                    asyncio.to_thread(agent.run, case.input, state), self.timeout_s)\n                stopped, response = result.stopped_reason, result.response\n                trace, final_state = result.trace, result.final_state\n            except Exception as e:\n                stopped, response = f\"error:{type(e).__name__}\", \"\"\n                trace, final_state = agent.tracer.trace, state.snapshot()\n            latency = (time.perf_counter() - t0) * 1000\n            return Run(case.id, trial, self.config.fingerprint(), response, trace,\n                       final_state, stopped, latency, estimate_cost(trace, self.config.model))\n\n    async def run(self, cases: list[Case]) -> list[Run]:\n        jobs = [self._run_one(c, t) for c in cases for t in range(self.trials)]\n        return await asyncio.gather(*jobs)\n```\n\nTwo details matter:\n\n- **Fresh state per run.** Each run gets its own copy of`initial_state` . Shared mutable state between concurrent runs is the most common source of flaky eval results.\n- **Rate limits.** Concurrency is bounded by a semaphore. Provider rate limits will otherwise turn into a wave of errors that look like agent failures.\n\n## The Report\n\n``` python\nfrom collections import defaultdict\n\ndef build_report(cases: list[Case], runs: list[Run], scorecard: Scorecard) -> dict:\n    by_id = {c.id: c for c in cases}\n    rows = []\n    for run in runs:\n        case = by_id[run.case_id]\n        result = scorecard.evaluate(case, run)\n        rows.append({\"case\": case, \"run\": run, **result})\n\n    def rate(rs): return sum(r[\"passed\"] for r in rs) / len(rs) if rs else 0.0\n\n    by_tag = defaultdict(list)\n    for r in rows:\n        for t in r[\"case\"].tags:\n            by_tag[t].append(r)\n\n    by_grader = defaultdict(list)\n    for r in rows:\n        for s in r[\"scores\"]:\n            by_grader[s.grader].append(s.passed)\n\n    return {\n        \"config\": runs[0].config_fingerprint,\n        \"n_runs\": len(rows),\n        \"pass_rate\": rate(rows),\n        \"by_tag\": {t: rate(rs) for t, rs in sorted(by_tag.items())},\n        \"by_grader\": {g: sum(v) / len(v) for g, v in by_grader.items()},\n        \"mean_cost_usd\": sum(r[\"run\"].cost_usd for r in rows) / len(rows),\n        \"p50_latency_ms\": sorted(r[\"run\"].latency_ms for r in rows)[len(rows) // 2],\n        \"failures\": [\n            {\"case\": r[\"case\"].id, \"trial\": r[\"run\"].trial,\n             \"reasons\": [s.reason for s in r[\"scores\"] if not s.passed],\n             \"trace_id\": r[\"run\"].trace.trace_id}\n            for r in rows if not r[\"passed\"]\n        ],\n    }\n```\n\nThe `failures` list is the most-read part of any report. Each entry links to a trace ID, so “why did case 47 fail?” is a single click, not a reproduction effort.\n\n## Build vs Buy\n\nYou don’t have to build all of this yourself. Here’s how hosted and open-source tools map onto the abstractions:\n\n| Abstraction | Examples | \n|---|---|\n| Case / dataset | LangSmith datasets, Braintrust datasets, Langfuse datasets, plain JSONL in git | \n| Runner | Inspect (UK AISI), OpenAI Evals, promptfoo, Braintrust `Eval()` , pytest | \n| Grader | DeepEval, Ragas, autoevals, your own functions | \n| Report | Braintrust, LangSmith experiments, Langfuse, a Markdown file posted to the PR | \n\nThe recommendation in this course: **own the Case, Grader, and Scorecard definitions in your repo**, because they encode your product’s definition of “correct.” Rent the runner, storage, and UI if it saves you time. If you swap vendors later, your test cases and graders shouldn’t have to change.\n\n## Key Takeaways\n\n- Evaluate at four layers: **component** ,**step** ,**trajectory** ,**outcome** . Outcome-only evals can’t explain regressions.\n- Five abstractions cover almost everything: **Case** ,**Run** ,**Grader** ,**Score** ,**Report**\n- Keep the grader contract narrow: `(Case, Run) → Score` , always with a reason\n- Separate **gates** (disqualifying failures) from**weighted** quality scores\n- Errors are results; give each run fresh state; bound concurrency\n- Own the definitions of correctness in your repo; rent infrastructure if it helps\n\n## Exercise\n\nImplement `Case`, `Run`, `Score`, `Scorecard`, `Runner`, and `build_report` for your agent. Write three trivial graders to prove the plumbing works:\n\n1. `CompletedGrader` : passes if`stopped_reason == \"final_answer\"`\n2. `MaxStepsGrader` : passes if the trace has at most 8 tool spans\n3. `NonEmptyResponseGrader` : passes if the response has at least 20 characters\n\nRun five cases with three trials each and print the report. Next lesson, you’ll replace those five cases with a real dataset.", "url": "https://wpnews.pro/news/designing-the-evaluation-framework", "canonical_source": "https://superml.org/tutorials/agent-eval-framework-design", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:05.582568+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Quill"], "also_reported_by": [], "takeaway": "Evaluate agents at all four layers — component, step, trajectory, and outcome — because an outcome-only harness cannot explain a drop from 88% to 81%, while step evals localize it to tool selection after get_invoices.", "recommendation": {"advice": "Add step-level evals that freeze message history at each decision point and assert the expected tool and arguments, so a regression can be traced to the specific decision rather than only the end-to-end score.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/designing-the-evaluation-framework", "markdown": "https://wpnews.pro/news/designing-the-evaluation-framework.md", "text": "https://wpnews.pro/news/designing-the-evaluation-framework.txt", "jsonld": "https://wpnews.pro/news/designing-the-evaluation-framework.jsonld"}}