# Capstone: Build an Agent Evaluation Framework

> Source: <https://superml.org/tutorials/agent-eval-capstone>
> Published: 2026-11-18 00:00:00+00:00

**Recommendation:** Calibrate each rubric judge against at least 50 human labels and record kappa and false-pass rate before gating CI, and include a deliberately broken sanity baseline agent to confirm the graders catch failures.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

· Agentic AI · 6 min read

### 📋 Prerequisites

- All previous lessons in Agent Evaluation & Architecture

### 🎯 What You'll Learn

- Design and implement a complete, layered evaluation framework for a real agent
- Produce a reliability report with confidence intervals, cost, and latency
- Ship a working CI regression gate and an online-to-offline feedback loop
- Write an evaluation design document that another engineer could maintain

## The Brief

You’ve built every component of an agent evaluation framework across twelve lessons. In this capstone, you assemble them into one system for one agent, and ship it the way you would at work: in a repository, with documentation, running in CI.

You have two options:

**Option A: Quill.** Build the billing-support agent used throughout the course, using a mock billing backend, then build its complete eval framework. Good if you want to focus entirely on evaluation.

**Option B: Your own agent.** Bring a tool-using agent you’ve built or maintain at work (sanitize anything confidential). Harder, and far more valuable: you’ll end up with a framework you actually use.

Either way, the deliverable is the same.

## Deliverables

```
agent-evals/
├── agent/                       # the agent, with the four seams (Lesson 2)
│   ├── loop.py
│   ├── model_client.py
│   ├── tools.py                 # live / mock / replay backends
│   └── state.py
├── config/
│   └── agent.yaml               # model, prompt version, temperature, max steps
├── prompts/
│   └── system_v1.md
├── evals/
│   ├── framework/               # Case, Run, Score, Scorecard, Runner, Report (Lesson 3)
│   ├── graders/
│   │   ├── state.py             # outcome + collateral changes (Lesson 5)
│   │   ├── policy.py            # gates (Lessons 5, 9)
│   │   ├── trajectory.py        # constraints, redundancy, step metrics (Lesson 7)
│   │   └── judges.py            # calibrated rubric judges (Lesson 6)
│   ├── datasets/
│   │   ├── quill.jsonl          # golden dataset with splits (Lesson 4)
│   │   ├── steps.jsonl          # step cases
│   │   └── conversations.jsonl  # multi-turn scenarios (Lesson 8)
│   ├── stats.py                 # pass^k, Wilson, bootstrap, paired test (Lesson 10)
│   ├── online.py                # reference-free evaluator + sampler (Lesson 11)
│   ├── run.py                   # CI entry point with gates (Lesson 12)
│   ├── JUDGES.md                # calibration records
│   └── tests/                   # unit tests for graders
├── .github/workflows/
│   ├── agent-evals.yml
│   └── eval-baseline.yml
└── EVAL_DESIGN.md               # the design document (template below)
```

## Requirements

### 1. Architecture (Lesson 2)

-  Every LLM and tool call produces a span; `run()` returns response, trace, final state, and stop reason
-  Tool backend supports **mock** and**replay** ; demonstrate replay reproducing a recorded failure
- Agent config is a versioned file with a fingerprint recorded on every run

### 2. Dataset (Lesson 4)

-  Coverage matrix with at least 12 meaningful cells, documented in `EVAL_DESIGN.md`
-  At least **120 single-turn cases** : at least 25 expert-authored edge cases, the rest synthetic or from real traffic
-  At least **40 step cases** and**10 multi-turn scenarios**
-  At least **30 adversarial cases** , including indirect injection through tool outputs, plus**15 benign look-alikes**
- Splits: smoke, regression, adversarial, holdout. Every case tagged by intent, difficulty, risk, and source.

### 3. Graders (Lessons 5–9)

- State outcome grader and no-collateral-changes grader
- At least three policy gates that read the trace
- Constraint-based trajectory grader, with redundancy and step-level metrics
-  At least two rubric judges, each **calibrated against at least 50 human labels** , with kappa and false-pass rate recorded in`JUDGES.md`
- Unit tests for every deterministic grader, each with at least three failing fixtures
-  A **sanity baseline** : a deliberately broken agent that your scorecard scores poorly

### 4. Reliability Report (Lesson 10)

- Regression split run with at least 3 trials
- pass@1 and pass^3 with cluster-bootstrap 95% confidence intervals
- Results sliced by intent and risk
- Flaky-case list with diagnosis of the top three
- Cost (mean, p95, per successful task) and latency (p50, p95)
- Attack success rate and over-refusal rate

### 5. Comparison Experiment

-  Compare **two configurations** (two models, two prompts, or a single model vs. a routed small/large setup)
- Paired comparison table with delta, confidence interval, and improved/regressed counts
- Read every regressed case; summarize what changed
- A one-paragraph recommendation: which configuration would you ship, and why?

### 6. CI (Lesson 12)

- Smoke and regression tiers running in GitHub Actions
- Hard gates, statistical regression gate (pass/warn/fail), slice gates, budget warnings
- Markdown report posted to the PR
- A demonstration PR that introduces a subtle regression and is blocked by the gate

### 7. Online Loop (Lesson 11)

- Reference-free online evaluator with stratified sampling
- Simulated or real production traffic run through it
-  At least **three flagged runs converted** into golden-dataset cases via`case_from_trace` , each confirmed to reproduce offline

## EVAL_DESIGN.md Template

The design document is what makes the framework maintainable by someone other than you. Use this structure:

```
# Evaluation Design: <agent name>

## 1. What the agent does
One paragraph. Users, tasks, tools, and what a side effect costs when it goes wrong.

## 2. Definition of success
Outcome, trajectory, and response quality, each defined concretely for this agent.

## 3. Threat model
Assets, threats, channels (table).

## 4. Coverage matrix
Dimensions, meaningful cells, target counts, actual counts.

## 5. Grader stack
Each grader: what it checks, gate or weighted (with weight), deterministic or judge.
Judges: link to JUDGES.md calibration results.

## 6. Splits and cadence
Which split runs when, trials per run, expected runtime and cost.

## 7. Gates
Hard gates, regression tolerance, slice tolerances, budget thresholds, override policy.

## 8. Online evaluation
Sampling strategy, real-time checks, alerts, review process, feedback loop.

## 9. Known gaps
What this framework doesn't measure yet, and what you'd add next.

## 10. Current results
Link to the latest reliability report.
```

Section 9 matters more than it looks. A framework that claims to measure everything is one nobody has read critically. Be specific about the blind spots.

## Suggested Schedule

| Session | Focus | Lessons | 
|---|---|---|
| 1 | Agent with seams; mock backend; first trace | 2 | 
| 2 | Framework abstractions; runner; first report | 3 | 
| 3 | Coverage matrix; dataset authoring and synthesis | 4 | 
| 4 | Deterministic graders and their tests | 5 | 
| 5 | Judges; human labeling; calibration | 6 | 
| 6 | Trajectory and step evals; multi-turn scenarios | 7, 8 | 
| 7 | Adversarial suite | 9 | 
| 8 | Reliability report and comparison experiment | 10 | 
| 9 | CI workflow and demonstration PR | 12 | 
| 10 | Online loop; EVAL_DESIGN.md; polish | 11 | 

## Self-Assessment Rubric

Grade yourself honestly. Each row is scored 0–3.

| Area | 0 | 1 | 2 | 3 | 
|---|---|---|---|---|
| **Architecture** | Final answer only | Traces exist | Traces + mock tools | All four seams + replay demonstrated | 
| **Dataset** | Demo queries | Untagged cases | Tagged, split | Coverage matrix met, sourced from multiple channels, holdout | 
| **Deterministic graders** | Keyword checks | Some state checks | State + policy gates | Full stack with tests and sanity baseline | 
| **Judges** | None or unvalidated | Rubric, no calibration | Calibrated, kappa reported | Calibrated, bias-tested, disagreements analyzed | 
| **Trajectory** | None | Exact-match path | Constraints | Constraints + step metrics + reasoning judge | 
| **Safety** | None | Direct prompts only | Plus permission gates | Plus indirect injection cross-product + over-refusal | 
| **Statistics** | Single run | Multiple trials | pass^k + CIs | Plus paired comparison and flaky-case diagnosis | 
| **CI** | Manual | Runs in CI | Gates block merge | Tiered, three-way verdict, PR report, override policy | 
| **Online loop** | None | Logging | Reference-free evaluators | Sampling + drift + cases flowing back into the dataset | 
| **Documentation** | None | README | Design doc | Design doc with honest known-gaps section | 

**24–30:** You’ve built a framework most production teams don’t have. Put it in your portfolio. **16–23:** Solid. Pick the two lowest rows and bring them to a 3. **Below 16:** Revisit the lessons for your lowest rows before calling it done.

## Presenting Your Work

If you’re using this capstone for a portfolio or an interview, the strongest artifact is a short write-up that tells one story: *“Here’s a regression my framework caught that a vibe check would have missed.”* Show the PR report, the regressed cases, the trace where behavior diverged, and the fix. That story demonstrates every skill in this course in five minutes.

## Where to Go Next

- **Production Agent Skills Engineering** applies these evaluation practices to reusable skill packs shared across many agents
- **Forward Deploy Engineer Mastery** shows how evaluation fits into shipping agentic workflows at customer sites
- Read the τ-bench, SWE-bench, and AgentBench papers to see how research benchmarks approach the same problems at scale, and where their methods differ from product evaluation

You started this course with the vibe check ceiling. If you’ve finished the capstone, you’re past it. Every change to your agent now comes with a number, a confidence interval, and a trace you can open.
