cd /news/ai-agents/capstone-build-an-agent-evaluation-f… Β· home β€Ί topics β€Ί ai-agents β€Ί article
[ARTICLE Β· art-172290] src=superml.org β†— pub= topic=ai-agents verified=true sentiment=Β· neutral

Capstone: Build an Agent Evaluation Framework

A capstone lesson in an agent evaluation course instructs engineers to assemble a complete, layered evaluation framework for a tool-using agent and ship it in a repository with CI. The deliverable requires at least 120 single-turn cases (including 25 expert-authored edge cases), 40 step cases, 10 multi-turn scenarios, 30 adversarial cases including indirect injection through tool outputs, and 15 benign look-alikes, plus at least two rubric judges each calibrated against at least 50 human labels with kappa and false-pass rate recorded in JUDGES.md. The framework must include a coverage matrix of at least 12 cells, smoke/regression/adversarial/holdout splits, a replay backend that reproduces a recorded failure, and a CI regression gate, with the goal of producing a reliability report with confidence intervals, cost, and latency.

recommendation

Calibrate each rubric judge against at least 50 human labels and record kappa and false-pass rate before gating CI, and include a deliberately broken sanity baseline agent to confirm the graders catch failures.

more in playbook: Evals and testing for LLM apps
read8 min views2 publishedNov 18, 2026
Capstone: Build an Agent Evaluation Framework
Image: Superml (auto-discovered)

Β· Agentic AI Β· 6 min read

πŸ“‹ Prerequisites

  • All previous lessons in Agent Evaluation & Architecture

🎯 What You'll Learn

  • Design and implement a complete, layered evaluation framework for a real agent
  • Produce a reliability report with confidence intervals, cost, and latency
  • Ship a working CI regression gate and an online-to-offline feedback loop
  • Write an evaluation design document that another engineer could maintain

The Brief #

You’ve built every component of an agent evaluation framework across twelve lessons. In this capstone, you assemble them into one system for one agent, and ship it the way you would at work: in a repository, with documentation, running in CI.

You have two options:

Option A: Quill. Build the billing-support agent used throughout the course, using a mock billing backend, then build its complete eval framework. Good if you want to focus entirely on evaluation.

Option B: Your own agent. Bring a tool-using agent you’ve built or maintain at work (sanitize anything confidential). Harder, and far more valuable: you’ll end up with a framework you actually use.

Either way, the deliverable is the same.

Deliverables #

agent-evals/
β”œβ”€β”€ agent/                       # the agent, with the four seams (Lesson 2)
β”‚   β”œβ”€β”€ loop.py
β”‚   β”œβ”€β”€ model_client.py
β”‚   β”œβ”€β”€ tools.py                 # live / mock / replay backends
β”‚   └── state.py
β”œβ”€β”€ config/
β”‚   └── agent.yaml               # model, prompt version, temperature, max steps
β”œβ”€β”€ prompts/
β”‚   └── system_v1.md
β”œβ”€β”€ evals/
β”‚   β”œβ”€β”€ framework/               # Case, Run, Score, Scorecard, Runner, Report (Lesson 3)
β”‚   β”œβ”€β”€ graders/
β”‚   β”‚   β”œβ”€β”€ state.py             # outcome + collateral changes (Lesson 5)
β”‚   β”‚   β”œβ”€β”€ policy.py            # gates (Lessons 5, 9)
β”‚   β”‚   β”œβ”€β”€ trajectory.py        # constraints, redundancy, step metrics (Lesson 7)
β”‚   β”‚   └── judges.py            # calibrated rubric judges (Lesson 6)
β”‚   β”œβ”€β”€ datasets/
β”‚   β”‚   β”œβ”€β”€ quill.jsonl          # golden dataset with splits (Lesson 4)
β”‚   β”‚   β”œβ”€β”€ steps.jsonl          # step cases
β”‚   β”‚   └── conversations.jsonl  # multi-turn scenarios (Lesson 8)
β”‚   β”œβ”€β”€ stats.py                 # pass^k, Wilson, bootstrap, paired test (Lesson 10)
β”‚   β”œβ”€β”€ online.py                # reference-free evaluator + sampler (Lesson 11)
β”‚   β”œβ”€β”€ run.py                   # CI entry point with gates (Lesson 12)
β”‚   β”œβ”€β”€ JUDGES.md                # calibration records
β”‚   └── tests/                   # unit tests for graders
β”œβ”€β”€ .github/workflows/
β”‚   β”œβ”€β”€ agent-evals.yml
β”‚   └── eval-baseline.yml
└── EVAL_DESIGN.md               # the design document (template below)

Requirements #

1. Architecture (Lesson 2)

  • Every LLM and tool call produces a span; run() returns response, trace, final state, and stop reason
  • Tool backend supports mock andreplay ; demonstrate replay reproducing a recorded failure
  • Agent config is a versioned file with a fingerprint recorded on every run

2. Dataset (Lesson 4)

  • Coverage matrix with at least 12 meaningful cells, documented in EVAL_DESIGN.md
  • At least 120 single-turn cases : at least 25 expert-authored edge cases, the rest synthetic or from real traffic
  • At least 40 step cases and10 multi-turn scenarios
  • At least 30 adversarial cases , including indirect injection through tool outputs, plus15 benign look-alikes
  • Splits: smoke, regression, adversarial, holdout. Every case tagged by intent, difficulty, risk, and source.

3. Graders (Lessons 5–9)

  • State outcome grader and no-collateral-changes grader
  • At least three policy gates that read the trace
  • Constraint-based trajectory grader, with redundancy and step-level metrics
  • At least two rubric judges, each calibrated against at least 50 human labels , with kappa and false-pass rate recorded inJUDGES.md
  • Unit tests for every deterministic grader, each with at least three failing fixtures
  • A sanity baseline : a deliberately broken agent that your scorecard scores poorly

4. Reliability Report (Lesson 10)

  • Regression split run with at least 3 trials
  • pass@1 and pass^3 with cluster-bootstrap 95% confidence intervals
  • Results sliced by intent and risk
  • Flaky-case list with diagnosis of the top three
  • Cost (mean, p95, per successful task) and latency (p50, p95)
  • Attack success rate and over-refusal rate

5. Comparison Experiment

  • Compare two configurations (two models, two prompts, or a single model vs. a routed small/large setup)
  • Paired comparison table with delta, confidence interval, and improved/regressed counts
  • Read every regressed case; summarize what changed
  • A one-paragraph recommendation: which configuration would you ship, and why?

6. CI (Lesson 12)

  • Smoke and regression tiers running in GitHub Actions
  • Hard gates, statistical regression gate (pass/warn/fail), slice gates, budget warnings
  • Markdown report posted to the PR
  • A demonstration PR that introduces a subtle regression and is blocked by the gate

7. Online Loop (Lesson 11)

  • Reference-free online evaluator with stratified sampling
  • Simulated or real production traffic run through it
  • At least three flagged runs converted into golden-dataset cases viacase_from_trace , each confirmed to reproduce offline

EVAL_DESIGN.md Template #

The design document is what makes the framework maintainable by someone other than you. Use this structure:


## 1. What the agent does
One paragraph. Users, tasks, tools, and what a side effect costs when it goes wrong.

## 2. Definition of success
Outcome, trajectory, and response quality, each defined concretely for this agent.

## 3. Threat model
Assets, threats, channels (table).

## 4. Coverage matrix
Dimensions, meaningful cells, target counts, actual counts.

## 5. Grader stack
Each grader: what it checks, gate or weighted (with weight), deterministic or judge.
Judges: link to JUDGES.md calibration results.

## 6. Splits and cadence
Which split runs when, trials per run, expected runtime and cost.

## 7. Gates
Hard gates, regression tolerance, slice tolerances, budget thresholds, override policy.

## 8. Online evaluation
Sampling strategy, real-time checks, alerts, review process, feedback loop.

## 9. Known gaps
What this framework doesn't measure yet, and what you'd add next.

## 10. Current results
Link to the latest reliability report.

Section 9 matters more than it looks. A framework that claims to measure everything is one nobody has read critically. Be specific about the blind spots.

Suggested Schedule #

Session Focus Lessons
1 Agent with seams; mock backend; first trace 2
2 Framework abstractions; runner; first report 3
3 Coverage matrix; dataset authoring and synthesis 4
4 Deterministic graders and their tests 5
5 Judges; human labeling; calibration 6
6 Trajectory and step evals; multi-turn scenarios 7, 8
7 Adversarial suite 9
8 Reliability report and comparison experiment 10
9 CI workflow and demonstration PR 12
10 Online loop; EVAL_DESIGN.md; polish 11

Self-Assessment Rubric #

Grade yourself honestly. Each row is scored 0–3.

Area 0 1 2 3
Architecture Final answer only Traces exist Traces + mock tools All four seams + replay demonstrated
Dataset Demo queries Untagged cases Tagged, split Coverage matrix met, sourced from multiple channels, holdout
Deterministic graders Keyword checks Some state checks State + policy gates Full stack with tests and sanity baseline
Judges None or unvalidated Rubric, no calibration Calibrated, kappa reported Calibrated, bias-tested, disagreements analyzed
Trajectory None Exact-match path Constraints Constraints + step metrics + reasoning judge
Safety None Direct prompts only Plus permission gates Plus indirect injection cross-product + over-refusal
Statistics Single run Multiple trials pass^k + CIs Plus paired comparison and flaky-case diagnosis
CI Manual Runs in CI Gates block merge Tiered, three-way verdict, PR report, override policy
Online loop None Logging Reference-free evaluators Sampling + drift + cases flowing back into the dataset
Documentation None README Design doc Design doc with honest known-gaps section

24–30: You’ve built a framework most production teams don’t have. Put it in your portfolio. 16–23: Solid. Pick the two lowest rows and bring them to a 3. Below 16: Revisit the lessons for your lowest rows before calling it done.

Presenting Your Work #

If you’re using this capstone for a portfolio or an interview, the strongest artifact is a short write-up that tells one story: β€œHere’s a regression my framework caught that a vibe check would have missed.” Show the PR report, the regressed cases, the trace where behavior diverged, and the fix. That story demonstrates every skill in this course in five minutes.

Where to Go Next #

  • Production Agent Skills Engineering applies these evaluation practices to reusable skill packs shared across many agents
  • Forward Deploy Engineer Mastery shows how evaluation fits into shipping agentic workflows at customer sites
  • Read the Ο„-bench, SWE-bench, and AgentBench papers to see how research benchmarks approach the same problems at scale, and where their methods differ from product evaluation

You started this course with the vibe check ceiling. If you’ve finished the capstone, you’re past it. Every change to your agent now comes with a number, a confidence interval, and a trace you can open.

── more in #ai-agents 4 stories Β· sorted by recency
── more on @quill 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/capstone-build-an-ag…] indexed:0 read:8min 2026-11-18 Β· β€”