# Designing the Evaluation Framework

> Source: <https://superml.org/tutorials/agent-eval-framework-design>
> Published: 2026-11-18 00:00:00+00:00

**Recommendation:** Add step-level evals that freeze message history at each decision point and assert the expected tool and arguments, so a regression can be traced to the specific decision rather than only the end-to-end score.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

· Agentic AI · 8 min read

### 📋 Prerequisites

- Eval-Driven Agent Architecture (previous lesson)

### 🎯 What You'll Learn

- Describe the four evaluation layers and what each one catches
- Implement the five core abstractions of an evaluation framework: Case, Run, Grader, Score, and Report
- Combine multiple graders into a weighted, gated scorecard
- Run an evaluation harness concurrently and produce a comparable report

## Why a Framework, Not a Script

Your first eval is usually a script: loop over some inputs, run the agent, check something, print a percentage. That’s the right way to start. It stops scaling as soon as you want to:

- Add a second kind of check without rewriting the loop
- Compare two agent versions on the same cases
- Run each case several times and report variance
- Find out *which* check failed on*which* case, and open its trace

An evaluation framework is the small set of abstractions that make all of that routine. It doesn’t need to be big. The version we build in this lesson is about 200 lines, and it’s the backbone of everything that follows.

## The Four Evaluation Layers

Agents fail at different granularities. A good framework evaluates at all four, because each layer catches failures the others miss.

```
 ┌────────────────────────────────────────────────────────────┐
 │ 4. OUTCOME     Did the task succeed end to end?            │  few, expensive, decisive
 ├────────────────────────────────────────────────────────────┤
 │ 3. TRAJECTORY  Was the path acceptable, safe, efficient?   │
 ├────────────────────────────────────────────────────────────┤
 │ 2. STEP        Was each individual decision correct?       │
 ├────────────────────────────────────────────────────────────┤
 │ 1. COMPONENT   Does each piece work in isolation?          │  many, cheap, diagnostic
 └────────────────────────────────────────────────────────────┘
```

### Layer 1: Component

Test pieces in isolation: a retrieval function returns the right KB article; a tool’s argument parser handles malformed dates; the intent classifier routes “I was charged twice” to billing. These are close to ordinary unit tests. They’re fast and they localize failures precisely.

### Layer 2: Step

Freeze the conversation at a specific point and ask: *given this exact context, does the model make the right next decision?* For Quill: given a message history where `get_invoices` just returned two identical charges, does the next call refund the duplicate rather than the original?

Step evals are the agent equivalent of unit tests for decisions. They’re cheap (one model call each), deterministic in their setup, and the best tool for regression-testing a specific bug fix.

```
step_case = {
    "messages": [...],               # history up to the decision point
    "expect_tool": "issue_refund",
    "expect_args": {"invoice_id": "INV-2291"},
}
```

### Layer 3: Trajectory

Run the full loop and grade the path: required tools called, forbidden tools avoided, correct ordering, no redundant calls, within a step budget. Covered in depth in Lesson 7.

### Layer 4: Outcome

Run the full loop and grade the end state and response. Did the right refund happen? Was the customer told accurately? This is the number stakeholders care about.

**The common mistake is having only Layer 4.** Your outcome score drops from 88% to 81%, and you have no idea why. With step and trajectory layers, you can see that tool-selection accuracy after `get_invoices` fell, which points directly at the prompt change that caused it.

## The Five Core Abstractions

```
 Case ──▶ Runner ──▶ Run ──▶ Grader(s) ──▶ Score(s) ──▶ Report
```

### Case

One test scenario: input, setup state, expectations, and metadata for slicing results.

``` python
from dataclasses import dataclass, field
from typing import Any

@dataclass(frozen=True)
class Case:
    id: str
    input: str
    initial_state: dict
    expected: dict = field(default_factory=dict)  # free-form, read by graders
    tags: tuple[str, ...] = ()                    # e.g. ("intent:duplicate", "risk:high")
    split: str = "regression"                     # "smoke" | "regression" | "adversarial"
```

`expected` is deliberately loose. Different graders read different keys: `expected["refund"]` for an outcome grader, `expected["forbidden_tools"]` for a trajectory grader, `expected["rubric_notes"]` for a judge.

### Run

The result of executing one case once. This is the `RunResult` from Lesson 2 plus identifying metadata.

```
@dataclass
class Run:
    case_id: str
    trial: int
    config_fingerprint: str
    response: str
    trace: "Trace"
    final_state: dict
    stopped_reason: str
    latency_ms: float
    cost_usd: float
```

### Grader

A function from `(Case, Run)` to a `Score`. That’s the entire contract, and keeping it that narrow is what lets you mix deterministic checks, trajectory scorers, and LLM judges freely.

``` python
from typing import Protocol

class Grader(Protocol):
    name: str
    def grade(self, case: Case, run: Run) -> "Score": ...
```

### Score

A grader’s verdict. Always include a reason. A score without a reason is unreadable when you’re debugging 30 failures at midnight.

```
@dataclass
class Score:
    grader: str
    value: float            # 0.0 to 1.0
    passed: bool
    reason: str = ""
    details: dict = field(default_factory=dict)
```

Normalize everything to 0–1 with a `passed` boolean. A judge’s 1–5 rating becomes `(rating - 1) / 4` with a pass threshold; a binary assertion is 0 or 1.

### Report

The aggregation over many runs: overall and per-grader pass rates, slices by tag, per-case details, cost, latency, and comparison against a baseline. Lesson 10 adds confidence intervals; Lesson 12 adds regression gates.

## Grader Composition: Gates and Weights

Not all graders are equal. Some failures are disqualifying regardless of everything else. For Quill, refunding over the policy limit is a failure even if the customer is delighted. Model this with two kinds of graders:

- **Gates** : must pass, or the run fails outright
- **Weighted** : contribute to a quality score

```
@dataclass
class Scorecard:
    gates: list[Grader]
    weighted: list[tuple[Grader, float]]
    pass_threshold: float = 0.7

    def evaluate(self, case: Case, run: Run) -> dict:
        gate_scores = [g.grade(case, run) for g in self.gates]
        weighted_scores = [(g.grade(case, run), w) for g, w in self.weighted]

        gates_ok = all(s.passed for s in gate_scores)
        total_w = sum(w for _, w in weighted_scores) or 1.0
        quality = sum(s.value * w for s, w in weighted_scores) / total_w

        return {
            "passed": gates_ok and quality >= self.pass_threshold,
            "quality": quality if gates_ok else 0.0,
            "scores": gate_scores + [s for s, _ in weighted_scores],
        }
```

A practical Quill scorecard:

| Grader | Type | Weight | 
|---|---|---|
| No refund over policy limit | Gate | — | 
| No action on wrong customer | Gate | — | 
| Completed (didn’t hit max steps) | Gate | — | 
| Correct final state | Weighted | 0.5 | 
| Trajectory acceptable | Weighted | 0.2 | 
| Response quality (judge) | Weighted | 0.3 | 

## The Runner

The runner executes cases against an agent factory, handling concurrency, repeated trials, timeouts, and errors. Errors are results, not crashes: an agent that throws is a failed run that should appear in the report.

``` python
import asyncio, time

class Runner:
    def __init__(self, agent_factory, config, trials: int = 1, concurrency: int = 8,
                 timeout_s: float = 120):
        self.agent_factory, self.config = agent_factory, config
        self.trials, self.timeout_s = trials, timeout_s
        self.sem = asyncio.Semaphore(concurrency)

    async def _run_one(self, case: Case, trial: int) -> Run:
        async with self.sem:
            agent, state = self.agent_factory(self.config, case.initial_state)
            t0 = time.perf_counter()
            try:
                result = await asyncio.wait_for(
                    asyncio.to_thread(agent.run, case.input, state), self.timeout_s)
                stopped, response = result.stopped_reason, result.response
                trace, final_state = result.trace, result.final_state
            except Exception as e:
                stopped, response = f"error:{type(e).__name__}", ""
                trace, final_state = agent.tracer.trace, state.snapshot()
            latency = (time.perf_counter() - t0) * 1000
            return Run(case.id, trial, self.config.fingerprint(), response, trace,
                       final_state, stopped, latency, estimate_cost(trace, self.config.model))

    async def run(self, cases: list[Case]) -> list[Run]:
        jobs = [self._run_one(c, t) for c in cases for t in range(self.trials)]
        return await asyncio.gather(*jobs)
```

Two details matter:

- **Fresh state per run.** Each run gets its own copy of`initial_state` . Shared mutable state between concurrent runs is the most common source of flaky eval results.
- **Rate limits.** Concurrency is bounded by a semaphore. Provider rate limits will otherwise turn into a wave of errors that look like agent failures.

## The Report

``` python
from collections import defaultdict

def build_report(cases: list[Case], runs: list[Run], scorecard: Scorecard) -> dict:
    by_id = {c.id: c for c in cases}
    rows = []
    for run in runs:
        case = by_id[run.case_id]
        result = scorecard.evaluate(case, run)
        rows.append({"case": case, "run": run, **result})

    def rate(rs): return sum(r["passed"] for r in rs) / len(rs) if rs else 0.0

    by_tag = defaultdict(list)
    for r in rows:
        for t in r["case"].tags:
            by_tag[t].append(r)

    by_grader = defaultdict(list)
    for r in rows:
        for s in r["scores"]:
            by_grader[s.grader].append(s.passed)

    return {
        "config": runs[0].config_fingerprint,
        "n_runs": len(rows),
        "pass_rate": rate(rows),
        "by_tag": {t: rate(rs) for t, rs in sorted(by_tag.items())},
        "by_grader": {g: sum(v) / len(v) for g, v in by_grader.items()},
        "mean_cost_usd": sum(r["run"].cost_usd for r in rows) / len(rows),
        "p50_latency_ms": sorted(r["run"].latency_ms for r in rows)[len(rows) // 2],
        "failures": [
            {"case": r["case"].id, "trial": r["run"].trial,
             "reasons": [s.reason for s in r["scores"] if not s.passed],
             "trace_id": r["run"].trace.trace_id}
            for r in rows if not r["passed"]
        ],
    }
```

The `failures` list is the most-read part of any report. Each entry links to a trace ID, so “why did case 47 fail?” is a single click, not a reproduction effort.

## Build vs Buy

You don’t have to build all of this yourself. Here’s how hosted and open-source tools map onto the abstractions:

| Abstraction | Examples | 
|---|---|
| Case / dataset | LangSmith datasets, Braintrust datasets, Langfuse datasets, plain JSONL in git | 
| Runner | Inspect (UK AISI), OpenAI Evals, promptfoo, Braintrust `Eval()` , pytest | 
| Grader | DeepEval, Ragas, autoevals, your own functions | 
| Report | Braintrust, LangSmith experiments, Langfuse, a Markdown file posted to the PR | 

The recommendation in this course: **own the Case, Grader, and Scorecard definitions in your repo**, because they encode your product’s definition of “correct.” Rent the runner, storage, and UI if it saves you time. If you swap vendors later, your test cases and graders shouldn’t have to change.

## Key Takeaways

- Evaluate at four layers: **component** ,**step** ,**trajectory** ,**outcome** . Outcome-only evals can’t explain regressions.
- Five abstractions cover almost everything: **Case** ,**Run** ,**Grader** ,**Score** ,**Report**
- Keep the grader contract narrow: `(Case, Run) → Score` , always with a reason
- Separate **gates** (disqualifying failures) from**weighted** quality scores
- Errors are results; give each run fresh state; bound concurrency
- Own the definitions of correctness in your repo; rent infrastructure if it helps

## Exercise

Implement `Case`, `Run`, `Score`, `Scorecard`, `Runner`, and `build_report` for your agent. Write three trivial graders to prove the plumbing works:

1. `CompletedGrader` : passes if`stopped_reason == "final_answer"`
2. `MaxStepsGrader` : passes if the trace has at most 8 tool spans
3. `NonEmptyResponseGrader` : passes if the response has at least 20 characters

Run five cases with three trials each and print the report. Next lesson, you’ll replace those five cases with a real dataset.
