Β· Agentic AI Β· 6 min read
π Prerequisites
- All previous lessons in Agent Evaluation & Architecture
π― What You'll Learn
- Design and implement a complete, layered evaluation framework for a real agent
- Produce a reliability report with confidence intervals, cost, and latency
- Ship a working CI regression gate and an online-to-offline feedback loop
- Write an evaluation design document that another engineer could maintain
The Brief #
Youβve built every component of an agent evaluation framework across twelve lessons. In this capstone, you assemble them into one system for one agent, and ship it the way you would at work: in a repository, with documentation, running in CI.
You have two options:
Option A: Quill. Build the billing-support agent used throughout the course, using a mock billing backend, then build its complete eval framework. Good if you want to focus entirely on evaluation.
Option B: Your own agent. Bring a tool-using agent youβve built or maintain at work (sanitize anything confidential). Harder, and far more valuable: youβll end up with a framework you actually use.
Either way, the deliverable is the same.
Deliverables #
agent-evals/
βββ agent/ # the agent, with the four seams (Lesson 2)
β βββ loop.py
β βββ model_client.py
β βββ tools.py # live / mock / replay backends
β βββ state.py
βββ config/
β βββ agent.yaml # model, prompt version, temperature, max steps
βββ prompts/
β βββ system_v1.md
βββ evals/
β βββ framework/ # Case, Run, Score, Scorecard, Runner, Report (Lesson 3)
β βββ graders/
β β βββ state.py # outcome + collateral changes (Lesson 5)
β β βββ policy.py # gates (Lessons 5, 9)
β β βββ trajectory.py # constraints, redundancy, step metrics (Lesson 7)
β β βββ judges.py # calibrated rubric judges (Lesson 6)
β βββ datasets/
β β βββ quill.jsonl # golden dataset with splits (Lesson 4)
β β βββ steps.jsonl # step cases
β β βββ conversations.jsonl # multi-turn scenarios (Lesson 8)
β βββ stats.py # pass^k, Wilson, bootstrap, paired test (Lesson 10)
β βββ online.py # reference-free evaluator + sampler (Lesson 11)
β βββ run.py # CI entry point with gates (Lesson 12)
β βββ JUDGES.md # calibration records
β βββ tests/ # unit tests for graders
βββ .github/workflows/
β βββ agent-evals.yml
β βββ eval-baseline.yml
βββ EVAL_DESIGN.md # the design document (template below)
Requirements #
1. Architecture (Lesson 2)
- Every LLM and tool call produces a span;
run()returns response, trace, final state, and stop reason - Tool backend supports mock andreplay ; demonstrate replay reproducing a recorded failure
- Agent config is a versioned file with a fingerprint recorded on every run
2. Dataset (Lesson 4)
- Coverage matrix with at least 12 meaningful cells, documented in
EVAL_DESIGN.md - At least 120 single-turn cases : at least 25 expert-authored edge cases, the rest synthetic or from real traffic
- At least 40 step cases and10 multi-turn scenarios
- At least 30 adversarial cases , including indirect injection through tool outputs, plus15 benign look-alikes
- Splits: smoke, regression, adversarial, holdout. Every case tagged by intent, difficulty, risk, and source.
3. Graders (Lessons 5β9)
- State outcome grader and no-collateral-changes grader
- At least three policy gates that read the trace
- Constraint-based trajectory grader, with redundancy and step-level metrics
- At least two rubric judges, each calibrated against at least 50 human labels , with kappa and false-pass rate recorded in
JUDGES.md - Unit tests for every deterministic grader, each with at least three failing fixtures
- A sanity baseline : a deliberately broken agent that your scorecard scores poorly
4. Reliability Report (Lesson 10)
- Regression split run with at least 3 trials
- pass@1 and pass^3 with cluster-bootstrap 95% confidence intervals
- Results sliced by intent and risk
- Flaky-case list with diagnosis of the top three
- Cost (mean, p95, per successful task) and latency (p50, p95)
- Attack success rate and over-refusal rate
5. Comparison Experiment
- Compare two configurations (two models, two prompts, or a single model vs. a routed small/large setup)
- Paired comparison table with delta, confidence interval, and improved/regressed counts
- Read every regressed case; summarize what changed
- A one-paragraph recommendation: which configuration would you ship, and why?
6. CI (Lesson 12)
- Smoke and regression tiers running in GitHub Actions
- Hard gates, statistical regression gate (pass/warn/fail), slice gates, budget warnings
- Markdown report posted to the PR
- A demonstration PR that introduces a subtle regression and is blocked by the gate
7. Online Loop (Lesson 11)
- Reference-free online evaluator with stratified sampling
- Simulated or real production traffic run through it
- At least three flagged runs converted into golden-dataset cases via
case_from_trace, each confirmed to reproduce offline
EVAL_DESIGN.md Template #
The design document is what makes the framework maintainable by someone other than you. Use this structure:
## 1. What the agent does
One paragraph. Users, tasks, tools, and what a side effect costs when it goes wrong.
## 2. Definition of success
Outcome, trajectory, and response quality, each defined concretely for this agent.
## 3. Threat model
Assets, threats, channels (table).
## 4. Coverage matrix
Dimensions, meaningful cells, target counts, actual counts.
## 5. Grader stack
Each grader: what it checks, gate or weighted (with weight), deterministic or judge.
Judges: link to JUDGES.md calibration results.
## 6. Splits and cadence
Which split runs when, trials per run, expected runtime and cost.
## 7. Gates
Hard gates, regression tolerance, slice tolerances, budget thresholds, override policy.
## 8. Online evaluation
Sampling strategy, real-time checks, alerts, review process, feedback loop.
## 9. Known gaps
What this framework doesn't measure yet, and what you'd add next.
## 10. Current results
Link to the latest reliability report.
Section 9 matters more than it looks. A framework that claims to measure everything is one nobody has read critically. Be specific about the blind spots.
Suggested Schedule #
| Session | Focus | Lessons |
|---|---|---|
| 1 | Agent with seams; mock backend; first trace | 2 |
| 2 | Framework abstractions; runner; first report | 3 |
| 3 | Coverage matrix; dataset authoring and synthesis | 4 |
| 4 | Deterministic graders and their tests | 5 |
| 5 | Judges; human labeling; calibration | 6 |
| 6 | Trajectory and step evals; multi-turn scenarios | 7, 8 |
| 7 | Adversarial suite | 9 |
| 8 | Reliability report and comparison experiment | 10 |
| 9 | CI workflow and demonstration PR | 12 |
| 10 | Online loop; EVAL_DESIGN.md; polish | 11 |
Self-Assessment Rubric #
Grade yourself honestly. Each row is scored 0β3.
| Area | 0 | 1 | 2 | 3 |
|---|---|---|---|---|
| Architecture | Final answer only | Traces exist | Traces + mock tools | All four seams + replay demonstrated |
| Dataset | Demo queries | Untagged cases | Tagged, split | Coverage matrix met, sourced from multiple channels, holdout |
| Deterministic graders | Keyword checks | Some state checks | State + policy gates | Full stack with tests and sanity baseline |
| Judges | None or unvalidated | Rubric, no calibration | Calibrated, kappa reported | Calibrated, bias-tested, disagreements analyzed |
| Trajectory | None | Exact-match path | Constraints | Constraints + step metrics + reasoning judge |
| Safety | None | Direct prompts only | Plus permission gates | Plus indirect injection cross-product + over-refusal |
| Statistics | Single run | Multiple trials | pass^k + CIs | Plus paired comparison and flaky-case diagnosis |
| CI | Manual | Runs in CI | Gates block merge | Tiered, three-way verdict, PR report, override policy |
| Online loop | None | Logging | Reference-free evaluators | Sampling + drift + cases flowing back into the dataset |
| Documentation | None | README | Design doc | Design doc with honest known-gaps section |
24β30: Youβve built a framework most production teams donβt have. Put it in your portfolio. 16β23: Solid. Pick the two lowest rows and bring them to a 3. Below 16: Revisit the lessons for your lowest rows before calling it done.
Presenting Your Work #
If youβre using this capstone for a portfolio or an interview, the strongest artifact is a short write-up that tells one story: βHereβs a regression my framework caught that a vibe check would have missed.β Show the PR report, the regressed cases, the trace where behavior diverged, and the fix. That story demonstrates every skill in this course in five minutes.
Where to Go Next #
- Production Agent Skills Engineering applies these evaluation practices to reusable skill packs shared across many agents
- Forward Deploy Engineer Mastery shows how evaluation fits into shipping agentic workflows at customer sites
- Read the Ο-bench, SWE-bench, and AgentBench papers to see how research benchmarks approach the same problems at scale, and where their methods differ from product evaluation
You started this course with the vibe check ceiling. If youβve finished the capstone, youβre past it. Every change to your agent now comes with a number, a confidence interval, and a trace you can open.