Capstone: Build an Agent Evaluation Framework A capstone lesson in an agent evaluation course instructs engineers to assemble a complete, layered evaluation framework for a tool-using agent and ship it in a repository with CI. The deliverable requires at least 120 single-turn cases (including 25 expert-authored edge cases), 40 step cases, 10 multi-turn scenarios, 30 adversarial cases including indirect injection through tool outputs, and 15 benign look-alikes, plus at least two rubric judges each calibrated against at least 50 human labels with kappa and false-pass rate recorded in JUDGES.md. The framework must include a coverage matrix of at least 12 cells, smoke/regression/adversarial/holdout splits, a replay backend that reproduces a recorded failure, and a CI regression gate, with the goal of producing a reliability report with confidence intervals, cost, and latency. Β· Agentic AI Β· 6 min read πŸ“‹ Prerequisites - All previous lessons in Agent Evaluation & Architecture 🎯 What You'll Learn - Design and implement a complete, layered evaluation framework for a real agent - Produce a reliability report with confidence intervals, cost, and latency - Ship a working CI regression gate and an online-to-offline feedback loop - Write an evaluation design document that another engineer could maintain The Brief You’ve built every component of an agent evaluation framework across twelve lessons. In this capstone, you assemble them into one system for one agent, and ship it the way you would at work: in a repository, with documentation, running in CI. You have two options: Option A: Quill. Build the billing-support agent used throughout the course, using a mock billing backend, then build its complete eval framework. Good if you want to focus entirely on evaluation. Option B: Your own agent. Bring a tool-using agent you’ve built or maintain at work sanitize anything confidential . Harder, and far more valuable: you’ll end up with a framework you actually use. Either way, the deliverable is the same. Deliverables agent-evals/ β”œβ”€β”€ agent/ the agent, with the four seams Lesson 2 β”‚ β”œβ”€β”€ loop.py β”‚ β”œβ”€β”€ model client.py β”‚ β”œβ”€β”€ tools.py live / mock / replay backends β”‚ └── state.py β”œβ”€β”€ config/ β”‚ └── agent.yaml model, prompt version, temperature, max steps β”œβ”€β”€ prompts/ β”‚ └── system v1.md β”œβ”€β”€ evals/ β”‚ β”œβ”€β”€ framework/ Case, Run, Score, Scorecard, Runner, Report Lesson 3 β”‚ β”œβ”€β”€ graders/ β”‚ β”‚ β”œβ”€β”€ state.py outcome + collateral changes Lesson 5 β”‚ β”‚ β”œβ”€β”€ policy.py gates Lessons 5, 9 β”‚ β”‚ β”œβ”€β”€ trajectory.py constraints, redundancy, step metrics Lesson 7 β”‚ β”‚ └── judges.py calibrated rubric judges Lesson 6 β”‚ β”œβ”€β”€ datasets/ β”‚ β”‚ β”œβ”€β”€ quill.jsonl golden dataset with splits Lesson 4 β”‚ β”‚ β”œβ”€β”€ steps.jsonl step cases β”‚ β”‚ └── conversations.jsonl multi-turn scenarios Lesson 8 β”‚ β”œβ”€β”€ stats.py pass^k, Wilson, bootstrap, paired test Lesson 10 β”‚ β”œβ”€β”€ online.py reference-free evaluator + sampler Lesson 11 β”‚ β”œβ”€β”€ run.py CI entry point with gates Lesson 12 β”‚ β”œβ”€β”€ JUDGES.md calibration records β”‚ └── tests/ unit tests for graders β”œβ”€β”€ .github/workflows/ β”‚ β”œβ”€β”€ agent-evals.yml β”‚ └── eval-baseline.yml └── EVAL DESIGN.md the design document template below Requirements 1. Architecture Lesson 2 - Every LLM and tool call produces a span; run returns response, trace, final state, and stop reason - Tool backend supports mock and replay ; demonstrate replay reproducing a recorded failure - Agent config is a versioned file with a fingerprint recorded on every run 2. Dataset Lesson 4 - Coverage matrix with at least 12 meaningful cells, documented in EVAL DESIGN.md - At least 120 single-turn cases : at least 25 expert-authored edge cases, the rest synthetic or from real traffic - At least 40 step cases and 10 multi-turn scenarios - At least 30 adversarial cases , including indirect injection through tool outputs, plus 15 benign look-alikes - Splits: smoke, regression, adversarial, holdout. Every case tagged by intent, difficulty, risk, and source. 3. Graders Lessons 5–9 - State outcome grader and no-collateral-changes grader - At least three policy gates that read the trace - Constraint-based trajectory grader, with redundancy and step-level metrics - At least two rubric judges, each calibrated against at least 50 human labels , with kappa and false-pass rate recorded in JUDGES.md - Unit tests for every deterministic grader, each with at least three failing fixtures - A sanity baseline : a deliberately broken agent that your scorecard scores poorly 4. Reliability Report Lesson 10 - Regression split run with at least 3 trials - pass@1 and pass^3 with cluster-bootstrap 95% confidence intervals - Results sliced by intent and risk - Flaky-case list with diagnosis of the top three - Cost mean, p95, per successful task and latency p50, p95 - Attack success rate and over-refusal rate 5. Comparison Experiment - Compare two configurations two models, two prompts, or a single model vs. a routed small/large setup - Paired comparison table with delta, confidence interval, and improved/regressed counts - Read every regressed case; summarize what changed - A one-paragraph recommendation: which configuration would you ship, and why? 6. CI Lesson 12 - Smoke and regression tiers running in GitHub Actions - Hard gates, statistical regression gate pass/warn/fail , slice gates, budget warnings - Markdown report posted to the PR - A demonstration PR that introduces a subtle regression and is blocked by the gate 7. Online Loop Lesson 11 - Reference-free online evaluator with stratified sampling - Simulated or real production traffic run through it - At least three flagged runs converted into golden-dataset cases via case from trace , each confirmed to reproduce offline EVAL DESIGN.md Template The design document is what makes the framework maintainable by someone other than you. Use this structure: Evaluation Design: