{"slug": "golden-datasets-and-scenario-suites", "title": "Golden Datasets and Scenario Suites", "summary": "A lesson on building evaluation datasets for AI agents argues that the golden dataset, not the grader, is the most important artifact in an evaluation framework because it defines what counts as correct behavior. The lesson recommends starting with a coverage matrix crossing intent, difficulty and risk dimensions — for the example customer-support agent Quill, intents include duplicate charge, refund request, invoice question, plan change, cancellation and out-of-scope — and weighting it toward high-risk and hard cases rather than production frequency. It also advises sourcing cases from stratified production traffic sampling, expert authoring and controlled synthetic generation, scrubbing PII and reconstructing fixture state from tool-call traces so scores stay comparable over time.", "body_md": "· Agentic AI · 8 min read\n\n### 📋 Prerequisites\n\n- Designing the Evaluation Framework (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Build a coverage matrix that defines what an eval set must contain\n- Source cases from production traffic, expert authoring, and controlled synthetic generation\n- Write case expectations that graders can check, with fixture state per case\n- Split, version, and maintain a dataset so scores stay comparable over time\n\n## The Dataset Is the Spec\n\nYour eval dataset is the most important artifact in your evaluation framework, more important than any grader. Graders decide *how* to score; the dataset decides *what counts*. A brilliant LLM judge running over fifteen demo queries tells you almost nothing.\n\nThink of the golden dataset as an executable specification of your agent’s job. If a behavior isn’t represented in the dataset, then as far as your process is concerned, the agent isn’t required to do it, and it will eventually stop doing it.\n\n## Start With a Coverage Matrix\n\nBefore writing a single case, decide what the dataset must cover. A coverage matrix crosses the dimensions that matter for your agent.\n\nFor Quill, three dimensions dominate:\n\n**Intent:** duplicate charge, refund request, invoice question, plan change, cancellation, out-of-scope, abusive/adversarial\n\n**Difficulty:** single-step lookup; multi-step with clear answer; ambiguous (needs clarification); conflicting data\n\n**Risk:** read-only; money movement under limit; money movement over limit (must escalate); account-level change\n\n```\nINTENTS = [\"duplicate_charge\", \"refund_request\", \"invoice_question\",\n           \"plan_change\", \"cancellation\", \"out_of_scope\"]\nDIFFICULTY = [\"simple\", \"multi_step\", \"ambiguous\", \"conflicting\"]\nRISK = [\"read_only\", \"refund_under_limit\", \"refund_over_limit\", \"account_change\"]\n\n# Not every combination is meaningful. Mark the ones that are, with a minimum count.\ncoverage_targets = {\n    (\"duplicate_charge\", \"simple\", \"refund_under_limit\"): 8,\n    (\"duplicate_charge\", \"conflicting\", \"refund_under_limit\"): 5,\n    (\"refund_request\", \"multi_step\", \"refund_over_limit\"): 6,\n    (\"invoice_question\", \"simple\", \"read_only\"): 6,\n    (\"cancellation\", \"ambiguous\", \"account_change\"): 4,\n    # ...\n}\n```\n\nThe matrix does two jobs. It tells you what to write, and later it tells you where you’re thin. A report that says “pass rate 91%” is less useful than one that says “pass rate 97% on simple cases and 64% on conflicting-data cases.” That second number is only possible if conflicting-data cases exist and are tagged.\n\n**Weight the matrix toward risk and difficulty, not frequency.** Production traffic might be 60% simple invoice questions, but those are the cases least likely to regress in a dangerous way. A good eval set over-samples the hard and high-stakes cells.\n\n## Three Sources of Cases\n\n### 1. Production Traffic (Best)\n\nReal user inputs capture phrasing, typos, mixed intents, and context you’d never invent. If you have logs or traces:\n\n1. **Sample stratified** , not randomly. Cluster inputs by intent (embedding clustering works well), then sample from each cluster.\n2. **Prioritize signal.** Over-sample conversations with thumbs-down feedback, escalations, retries, long trajectories, and tool errors.\n3. **Scrub PII** before anything enters the dataset. Replace names, emails, and account numbers with consistent synthetic values.\n4. **Reconstruct state.** A production input is only a test case once you know what the world looked like at the time: which invoices existed, what the customer’s plan was. Capture this from the trace’s tool outputs.\n\n``` php\ndef case_from_trace(trace, case_id: str, scrub) -> dict:\n    \"\"\"Turn a production trace into a draft case. A human still writes 'expected'.\"\"\"\n    user_input = scrub(trace.spans[0].input)\n    state = {\"customers\": {}, \"invoices\": {}}\n    for s in trace.of_kind(\"tool\"):\n        if s.name == \"lookup_customer\" and \"id\" in (s.output or {}):\n            c = scrub(s.output); state[\"customers\"][c[\"id\"]] = c\n        if s.name == \"get_invoices\":\n            for inv in s.output or []:\n                inv = scrub(inv); state[\"invoices\"][inv[\"id\"]] = inv\n    return {\"id\": case_id, \"input\": user_input, \"initial_state\": state,\n            \"expected\": {}, \"tags\": [], \"source\": f\"prod:{trace.trace_id}\"}\n```\n\n### 2. Expert Authoring (Essential for Edge Cases)\n\nSome cases are rare in production but critical: the customer with two accounts under similar emails, the refund that’s exactly at the $200 limit, the invoice that was already refunded once. Domain experts (support leads, in Quill’s case) should write these. They know the failure modes that matter and which ones cost money.\n\nA useful prompt for the expert session: *“Tell me about the last five tickets that went wrong, and the ones new support reps always get wrong.”*\n\n### 3. Synthetic Generation (Controlled Volume)\n\nLLMs are good at generating varied phrasings and filling matrix cells. They’re bad at generating realistic *difficulty*: synthetic cases tend to be cleaner, more explicit, and easier than real ones. Use synthesis to fill coverage gaps, not as your primary source.\n\nThe pattern that works: **generate the scenario and state programmatically, use the LLM only for the user’s message.**\n\n``` php\nimport random\n\ndef synth_duplicate_charge(rng: random.Random, llm) -> dict:\n    cid = f\"C{rng.randint(1000, 9999)}\"\n    amount = rng.choice([19.0, 49.0, 99.0, 149.0])\n    month = rng.choice([\"January\", \"February\", \"March\", \"April\"])\n    inv_a, inv_b = f\"INV-{rng.randint(1000, 9999)}\", f\"INV-{rng.randint(1000, 9999)}\"\n\n    state = {\n        \"customers\": {cid: {\"id\": cid, \"email\": f\"user{cid}@example.com\", \"plan\": \"pro\"}},\n        \"invoices\": {\n            inv_a: {\"id\": inv_a, \"customer_id\": cid, \"amount\": amount, \"month\": month, \"status\": \"paid\"},\n            inv_b: {\"id\": inv_b, \"customer_id\": cid, \"amount\": amount, \"month\": month, \"status\": \"paid\"},\n        },\n    }\n    persona = rng.choice([\"terse\", \"frustrated\", \"polite and verbose\", \"non-native English speaker\"])\n    message = llm(\n        f\"Write a support message from a {persona} customer (email user{cid}@example.com) \"\n        f\"who was charged ${amount:.0f} twice in {month}. Don't mention invoice IDs. \"\n        f\"Output only the message.\"\n    )\n    return {\n        \"input\": message,\n        \"initial_state\": state,\n        \"expected\": {\n            \"refund\": {\"invoice_id_in\": [inv_a, inv_b], \"amount\": amount, \"count\": 1},\n            \"forbidden_tools\": [],\n        },\n        \"tags\": [\"intent:duplicate_charge\", \"difficulty:simple\", \"risk:refund_under_limit\",\n                 f\"persona:{persona}\", \"source:synthetic\"],\n    }\n```\n\nBecause the state and expectation are constructed in code, they’re guaranteed consistent. The LLM can’t invent an invoice that doesn’t exist in the fixture. Always **review a sample of synthetic cases by hand** before adding them; generators drift into repetitive phrasings.\n\n## Writing Good Expectations\n\nA case is only as good as what a grader can check. Weak expectations produce weak evals.\n\n| Weak | Strong | \n|---|---|\n| `\"expected_answer\": \"Refunded the duplicate charge\"` | `\"refund\": {\"invoice_id_in\": [\"INV-1\", \"INV-2\"], \"amount\": 49.0, \"count\": 1}` | \n| `\"should_escalate\": true` | `\"escalation\": {\"required\": true, \"priority_in\": [\"high\", \"urgent\"]}, \"refund\": {\"count\": 0}` | \n| `\"good_response\": true` | `\"must_mention\": [\"5-7 business days\"], \"must_not_claim\": [\"refund is instant\"]` | \n\nRules of thumb:\n\n- **Express outcomes as state.** “A refund exists with these properties” beats “the response says it refunded.”\n- **Allow valid alternatives.**`invoice_id_in` accepts either duplicate. Over-specified expectations create false failures, and a team that sees false failures stops trusting the eval.\n- **Specify negatives.** What must*not* happen is often more important than what must. “No refund” is an expectation.\n- **Leave room for judgment.** For response quality, write`rubric_notes` that a judge reads (“should acknowledge the frustration; should not promise a timeline shorter than 5 business days”), not a reference answer to match.\n\n## Storage Format\n\nJSONL in git is the right default. It diffs cleanly, reviews in pull requests, and every eval run can record the exact commit of the dataset it used.\n\n```\n{\"id\": \"dup-0007\", \"input\": \"hey i got billed twice for march?? 49 bucks each\", \"initial_state\": {\"customers\": {\"C4410\": {\"id\": \"C4410\", \"email\": \"dana@example.com\", \"plan\": \"pro\"}}, \"invoices\": {\"INV-2290\": {\"id\": \"INV-2290\", \"customer_id\": \"C4410\", \"amount\": 49.0, \"month\": \"March\", \"status\": \"paid\"}, \"INV-2291\": {\"id\": \"INV-2291\", \"customer_id\": \"C4410\", \"amount\": 49.0, \"month\": \"March\", \"status\": \"paid\"}}}, \"expected\": {\"refund\": {\"invoice_id_in\": [\"INV-2290\", \"INV-2291\"], \"amount\": 49.0, \"count\": 1}}, \"tags\": [\"intent:duplicate_charge\", \"difficulty:simple\", \"risk:refund_under_limit\", \"source:prod\"], \"split\": \"regression\"}\n```\n\nLarge fixtures (documents, database dumps) can live in a separate directory referenced by path.\n\n## Splits\n\nNot every case runs every time. Split the dataset by purpose:\n\n| Split | Size | When it runs | Purpose | \n|---|---|---|---|\n| **smoke** | 15–30 | Every commit, in minutes | Catch catastrophic breakage | \n| **regression** | 150–500 | Every PR / nightly | The main quality number | \n| **adversarial** | 50–200 | Nightly and before release | Safety and abuse (Lesson 9) | \n| **holdout** | 10–20% of regression | Rarely, before major releases | Detect overfitting to the eval set | \n\nThe **holdout** deserves a word. If your team tunes prompts against the regression set for months, the agent will start fitting those specific cases. A holdout set that nobody looks at during development tells you whether improvements generalize. If regression goes up and holdout stays flat, you’re overfitting.\n\n## How Many Cases?\n\nFewer than you fear, more than you have. Lesson 10 covers the statistics properly, but as a rough guide: to detect a 5-point change in pass rate with reasonable confidence, you need a few hundred cases (or fewer cases with several trials each). For a single coverage-matrix cell, 5–10 cases is enough to notice a collapse, not to measure small changes.\n\nStart with 50 good cases. A small, well-designed dataset beats a large sloppy one, because every failure in it is worth reading.\n\n## Versioning and Maintenance\n\nA dataset is living code. Treat it that way:\n\n- **Version it.** Every report records the dataset version (a git SHA or semantic version). Scores from different versions aren’t comparable.\n- **Never silently edit a case.** If an expectation was wrong, fix it in a reviewed commit with a reason. Changing expectations to make the agent pass is the eval equivalent of deleting a failing test.\n- **Grow from failures.** Every production incident and every bug report becomes at least one new case (Lesson 11 automates this).\n- **Prune duplicates.** Near-identical cases inflate some cells and distort pass rates. Deduplicate periodically by embedding similarity.\n- **Re-audit labels.** Policies change. When the refund limit moves from $200 to $250, every case that depends on it must be updated in the same change.\n\n## Key Takeaways\n\n- The dataset is the executable spec of your agent’s job. Invest in it more than in graders.\n- Start from a **coverage matrix** over intent, difficulty, and risk; over-sample hard and high-risk cells\n- Source from **production** (realism),**experts** (edge cases), and**synthesis** (coverage), in that order of value\n- Generate state and expectations in code; use the LLM only for natural-language variation\n- Express expectations as **state** , allow valid alternatives, and specify negatives\n- Split into **smoke** ,**regression** ,**adversarial** , and**holdout** ; version everything\n\n## Exercise\n\nBuild a 50-case golden dataset for Quill (or your agent):\n\n1. Write a coverage matrix with at least 12 meaningful cells\n2. Author 15 edge cases by hand\n3. Generate 35 cases with a synthetic generator that builds state in code\n4. Tag every case with intent, difficulty, risk, and source\n5. Mark 15 cases as `smoke`\n\nCommit it as `evals/datasets/quill-v1.jsonl`. The next lesson writes the graders that check it.", "url": "https://wpnews.pro/news/golden-datasets-and-scenario-suites", "canonical_source": "https://superml.org/tutorials/agent-eval-golden-datasets", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:10.418725+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools"], "entities": ["Quill"], "also_reported_by": [], "takeaway": "Build a coverage matrix that tags each eval case by intent, difficulty and risk with minimum counts per cell, so reports can show pass rates like 97% on simple cases versus 64% on conflicting-data cases instead of a single aggregate.", "recommendation": {"advice": "Define a coverage matrix of intent, difficulty and risk cells with minimum case counts, over-sample high-risk and conflicting-data cells, and tag every case so eval reports break down pass rates by cell.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/golden-datasets-and-scenario-suites", "markdown": "https://wpnews.pro/news/golden-datasets-and-scenario-suites.md", "text": "https://wpnews.pro/news/golden-datasets-and-scenario-suites.txt", "jsonld": "https://wpnews.pro/news/golden-datasets-and-scenario-suites.jsonld"}}