Golden Datasets and Scenario Suites A lesson on building evaluation datasets for AI agents argues that the golden dataset, not the grader, is the most important artifact in an evaluation framework because it defines what counts as correct behavior. The lesson recommends starting with a coverage matrix crossing intent, difficulty and risk dimensions — for the example customer-support agent Quill, intents include duplicate charge, refund request, invoice question, plan change, cancellation and out-of-scope — and weighting it toward high-risk and hard cases rather than production frequency. It also advises sourcing cases from stratified production traffic sampling, expert authoring and controlled synthetic generation, scrubbing PII and reconstructing fixture state from tool-call traces so scores stay comparable over time. · Agentic AI · 8 min read 📋 Prerequisites - Designing the Evaluation Framework previous lesson 🎯 What You'll Learn - Build a coverage matrix that defines what an eval set must contain - Source cases from production traffic, expert authoring, and controlled synthetic generation - Write case expectations that graders can check, with fixture state per case - Split, version, and maintain a dataset so scores stay comparable over time The Dataset Is the Spec Your eval dataset is the most important artifact in your evaluation framework, more important than any grader. Graders decide how to score; the dataset decides what counts . A brilliant LLM judge running over fifteen demo queries tells you almost nothing. Think of the golden dataset as an executable specification of your agent’s job. If a behavior isn’t represented in the dataset, then as far as your process is concerned, the agent isn’t required to do it, and it will eventually stop doing it. Start With a Coverage Matrix Before writing a single case, decide what the dataset must cover. A coverage matrix crosses the dimensions that matter for your agent. For Quill, three dimensions dominate: Intent: duplicate charge, refund request, invoice question, plan change, cancellation, out-of-scope, abusive/adversarial Difficulty: single-step lookup; multi-step with clear answer; ambiguous needs clarification ; conflicting data Risk: read-only; money movement under limit; money movement over limit must escalate ; account-level change INTENTS = "duplicate charge", "refund request", "invoice question", "plan change", "cancellation", "out of scope" DIFFICULTY = "simple", "multi step", "ambiguous", "conflicting" RISK = "read only", "refund under limit", "refund over limit", "account change" Not every combination is meaningful. Mark the ones that are, with a minimum count. coverage targets = { "duplicate charge", "simple", "refund under limit" : 8, "duplicate charge", "conflicting", "refund under limit" : 5, "refund request", "multi step", "refund over limit" : 6, "invoice question", "simple", "read only" : 6, "cancellation", "ambiguous", "account change" : 4, ... } The matrix does two jobs. It tells you what to write, and later it tells you where you’re thin. A report that says “pass rate 91%” is less useful than one that says “pass rate 97% on simple cases and 64% on conflicting-data cases.” That second number is only possible if conflicting-data cases exist and are tagged. Weight the matrix toward risk and difficulty, not frequency. Production traffic might be 60% simple invoice questions, but those are the cases least likely to regress in a dangerous way. A good eval set over-samples the hard and high-stakes cells. Three Sources of Cases 1. Production Traffic Best Real user inputs capture phrasing, typos, mixed intents, and context you’d never invent. If you have logs or traces: 1. Sample stratified , not randomly. Cluster inputs by intent embedding clustering works well , then sample from each cluster. 2. Prioritize signal. Over-sample conversations with thumbs-down feedback, escalations, retries, long trajectories, and tool errors. 3. Scrub PII before anything enters the dataset. Replace names, emails, and account numbers with consistent synthetic values. 4. Reconstruct state. A production input is only a test case once you know what the world looked like at the time: which invoices existed, what the customer’s plan was. Capture this from the trace’s tool outputs. php def case from trace trace, case id: str, scrub - dict: """Turn a production trace into a draft case. A human still writes 'expected'.""" user input = scrub trace.spans 0 .input state = {"customers": {}, "invoices": {}} for s in trace.of kind "tool" : if s.name == "lookup customer" and "id" in s.output or {} : c = scrub s.output ; state "customers" c "id" = c if s.name == "get invoices": for inv in s.output or : inv = scrub inv ; state "invoices" inv "id" = inv return {"id": case id, "input": user input, "initial state": state, "expected": {}, "tags": , "source": f"prod:{trace.trace id}"} 2. Expert Authoring Essential for Edge Cases Some cases are rare in production but critical: the customer with two accounts under similar emails, the refund that’s exactly at the $200 limit, the invoice that was already refunded once. Domain experts support leads, in Quill’s case should write these. They know the failure modes that matter and which ones cost money. A useful prompt for the expert session: “Tell me about the last five tickets that went wrong, and the ones new support reps always get wrong.” 3. Synthetic Generation Controlled Volume LLMs are good at generating varied phrasings and filling matrix cells. They’re bad at generating realistic difficulty : synthetic cases tend to be cleaner, more explicit, and easier than real ones. Use synthesis to fill coverage gaps, not as your primary source. The pattern that works: generate the scenario and state programmatically, use the LLM only for the user’s message. php import random def synth duplicate charge rng: random.Random, llm - dict: cid = f"C{rng.randint 1000, 9999 }" amount = rng.choice 19.0, 49.0, 99.0, 149.0 month = rng.choice "January", "February", "March", "April" inv a, inv b = f"INV-{rng.randint 1000, 9999 }", f"INV-{rng.randint 1000, 9999 }" state = { "customers": {cid: {"id": cid, "email": f"user{cid}@example.com", "plan": "pro"}}, "invoices": { inv a: {"id": inv a, "customer id": cid, "amount": amount, "month": month, "status": "paid"}, inv b: {"id": inv b, "customer id": cid, "amount": amount, "month": month, "status": "paid"}, }, } persona = rng.choice "terse", "frustrated", "polite and verbose", "non-native English speaker" message = llm f"Write a support message from a {persona} customer email user{cid}@example.com " f"who was charged ${amount:.0f} twice in {month}. Don't mention invoice IDs. " f"Output only the message." return { "input": message, "initial state": state, "expected": { "refund": {"invoice id in": inv a, inv b , "amount": amount, "count": 1}, "forbidden tools": , }, "tags": "intent:duplicate charge", "difficulty:simple", "risk:refund under limit", f"persona:{persona}", "source:synthetic" , } Because the state and expectation are constructed in code, they’re guaranteed consistent. The LLM can’t invent an invoice that doesn’t exist in the fixture. Always review a sample of synthetic cases by hand before adding them; generators drift into repetitive phrasings. Writing Good Expectations A case is only as good as what a grader can check. Weak expectations produce weak evals. | Weak | Strong | |---|---| | "expected answer": "Refunded the duplicate charge" | "refund": {"invoice id in": "INV-1", "INV-2" , "amount": 49.0, "count": 1} | | "should escalate": true | "escalation": {"required": true, "priority in": "high", "urgent" }, "refund": {"count": 0} | | "good response": true | "must mention": "5-7 business days" , "must not claim": "refund is instant" | Rules of thumb: - Express outcomes as state. “A refund exists with these properties” beats “the response says it refunded.” - Allow valid alternatives. invoice id in accepts either duplicate. Over-specified expectations create false failures, and a team that sees false failures stops trusting the eval. - Specify negatives. What must not happen is often more important than what must. “No refund” is an expectation. - Leave room for judgment. For response quality, write rubric notes that a judge reads “should acknowledge the frustration; should not promise a timeline shorter than 5 business days” , not a reference answer to match. Storage Format JSONL in git is the right default. It diffs cleanly, reviews in pull requests, and every eval run can record the exact commit of the dataset it used. {"id": "dup-0007", "input": "hey i got billed twice for march?? 49 bucks each", "initial state": {"customers": {"C4410": {"id": "C4410", "email": "dana@example.com", "plan": "pro"}}, "invoices": {"INV-2290": {"id": "INV-2290", "customer id": "C4410", "amount": 49.0, "month": "March", "status": "paid"}, "INV-2291": {"id": "INV-2291", "customer id": "C4410", "amount": 49.0, "month": "March", "status": "paid"}}}, "expected": {"refund": {"invoice id in": "INV-2290", "INV-2291" , "amount": 49.0, "count": 1}}, "tags": "intent:duplicate charge", "difficulty:simple", "risk:refund under limit", "source:prod" , "split": "regression"} Large fixtures documents, database dumps can live in a separate directory referenced by path. Splits Not every case runs every time. Split the dataset by purpose: | Split | Size | When it runs | Purpose | |---|---|---|---| | smoke | 15–30 | Every commit, in minutes | Catch catastrophic breakage | | regression | 150–500 | Every PR / nightly | The main quality number | | adversarial | 50–200 | Nightly and before release | Safety and abuse Lesson 9 | | holdout | 10–20% of regression | Rarely, before major releases | Detect overfitting to the eval set | The holdout deserves a word. If your team tunes prompts against the regression set for months, the agent will start fitting those specific cases. A holdout set that nobody looks at during development tells you whether improvements generalize. If regression goes up and holdout stays flat, you’re overfitting. How Many Cases? Fewer than you fear, more than you have. Lesson 10 covers the statistics properly, but as a rough guide: to detect a 5-point change in pass rate with reasonable confidence, you need a few hundred cases or fewer cases with several trials each . For a single coverage-matrix cell, 5–10 cases is enough to notice a collapse, not to measure small changes. Start with 50 good cases. A small, well-designed dataset beats a large sloppy one, because every failure in it is worth reading. Versioning and Maintenance A dataset is living code. Treat it that way: - Version it. Every report records the dataset version a git SHA or semantic version . Scores from different versions aren’t comparable. - Never silently edit a case. If an expectation was wrong, fix it in a reviewed commit with a reason. Changing expectations to make the agent pass is the eval equivalent of deleting a failing test. - Grow from failures. Every production incident and every bug report becomes at least one new case Lesson 11 automates this . - Prune duplicates. Near-identical cases inflate some cells and distort pass rates. Deduplicate periodically by embedding similarity. - Re-audit labels. Policies change. When the refund limit moves from $200 to $250, every case that depends on it must be updated in the same change. Key Takeaways - The dataset is the executable spec of your agent’s job. Invest in it more than in graders. - Start from a coverage matrix over intent, difficulty, and risk; over-sample hard and high-risk cells - Source from production realism , experts edge cases , and synthesis coverage , in that order of value - Generate state and expectations in code; use the LLM only for natural-language variation - Express expectations as state , allow valid alternatives, and specify negatives - Split into smoke , regression , adversarial , and holdout ; version everything Exercise Build a 50-case golden dataset for Quill or your agent : 1. Write a coverage matrix with at least 12 meaningful cells 2. Author 15 edge cases by hand 3. Generate 35 cases with a synthetic generator that builds state in code 4. Tag every case with intent, difficulty, risk, and source 5. Mark 15 cases as smoke Commit it as evals/datasets/quill-v1.jsonl . The next lesson writes the graders that check it.