{"slug": "how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs", "title": "How to Build a Golden Dataset for AI Agent Evaluation From Your Logs", "summary": "Teams building AI agent evaluations should replace hand-written test cases with a golden dataset harvested from production traces, logging full trajectories including tool calls, token cost and outcome signals, according to guidance citing Anthropic's engineering team's advice to start with roughly 20 to 50 tasks drawn from real failures. The article recommends sampling failures deliberately via tool errors, loops, cost outliers and thumbs-downs while adding a quarter to a third of ordinary successes, labeling end states rather than wording, and sizing the set to the decision at hand — at a 90% pass rate, 100 cases carry about plus or minus 6 points of noise. It also advises keeping a frozen regression set that grows with every incident alongside a rolling fresh set relabelled monthly, and scrubbing personal data before copying traces into a dataset.", "body_md": "*Your 20 hand-written test cases describe the product you imagined in March, not the one customers use now. The traces you already store can fix that in about a week of part-time work.*\n\n- Start from a golden dataset built out of production logs: log full trajectories with tool calls and outcome signals, then scrub personal data before copying anything.\n- Sample failures on purpose using tool errors, loops, cost outliers and thumbs-downs, but add a quarter to a third of ordinary successes so the set can catch regressions.\n- Label the end state, such as the database row or ticket status, rather than the wording, and let a model draft labels that a person confirms.\n- At a 90% pass rate, 100 cases leave about plus or minus 6 points of noise, so size the set by the decision you need to make and run each case several times.\n- Keep two tiers, a frozen regression set that grows with every incident and a rolling fresh set that is relabelled monthly.\n\nHand-written test cases start going stale the day you write them, which is why how to build a golden dataset for AI agent evaluation comes down to one move: stop inventing inputs and start harvesting them. Your agent has already been run by real users thousands of times. Every run left a trace, and those traces know things your imagination doesn't.\n\nThe usual setup looks the same almost everywhere. Someone writes 20 prompts the week before launch, every one a clean request the agent handles well, wires them into CI and watches a green number. Six months later the agent has a new tool, a new model and a customer base asking things nobody predicted, and the suite still passes at 95%. That number isn't measuring your product anymore.\n\nAnthropic's engineering team has advised teams building agent evals to start small, with something like 20 to 50 tasks drawn from real failures, rather than waiting for a perfect suite. That's the right size to begin with. What follows is where those tasks come from, how to pick them, how big the set has to be, and how to keep it from rotting.\n\nA golden dataset from production logs is only as good as what you logged. For an agent, the final answer isn't enough. You need the full trajectory: the user's input, every model call, each tool call with its arguments and return value, the step count, latency, token cost, and the version of the prompt and model that ran. LangSmith, Langfuse, Braintrust and Arize Phoenix all capture this and can push selected traces into a dataset. If you'd rather not depend on a vendor, OpenTelemetry's GenAI semantic conventions give you a common schema to write to yourself.\n\n[How to calibrate LLM as a judge before your eval dashboard lies to you](https://startupfortune.com/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you/)\n\nIn the article's worked example, a judge that agrees with humans on 86% of 100 outputs scores a Cohen's kappa of only 0.46, because it fails just 8 of the 20 bad outputs. Correcting its 90% pass rate gives 80%, the human figure. - [how to calibrate LLM as a judge](https://startupfortune.com/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you/) - [validate LLM judge against human ratings](https://startupfortune.com/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you/)\n\nLog the outcome signals too. A thumbs-down, a user rephrasing the same request within a minute, a handoff to a human, a refund, a ticket reopened the next day. These are free labels, and they're the cheapest way to find the traces worth your time.\n\nScrub before you copy. Redact emails, names, account numbers and any API keys users paste into prompts, then store the dataset somewhere with tighter access than your raw logs. A golden set lives for years and gets passed to contractors and vendors, so it shouldn't contain anything you wouldn't hand them.\n\nOne agent-specific trap deserves its own warning. A production trace was recorded against a world that no longer exists: the customer's order has shipped, the calendar slot is taken, the inventory count changed. If you replay the input against live systems, the agent will do something different and your label will be wrong through no fault of the agent. Snapshot the relevant state with each case, or point the agent at mocked or sandboxed tools seeded with that snapshot. It's tedious, and it's the step most teams skip until their first confusing eval run.\n\n## Sample failures on purpose, then add the boring successes\n\nRandom sampling wastes your reading time. If the agent succeeds 90% of the time, a random batch of 100 traces hands you about ten failures and ninety cases that teach you nothing. Rank traces by trouble instead. Tool calls that returned errors, runs whose step count sits above the 95th percentile, cost outliers, negative feedback, escalations to a human, and low scores from an automated grader are all good filters. Pull from the top of each list.\n\nThen do the part people forget. A set made only of failures can't tell you that a prompt tweak broke something that used to work. Add a random slice of ordinary successes, somewhere between a quarter and a third of the set, so a regression in the common path shows up instead of hiding.\n\nNext, the long tail. Embed the user inputs, cluster them, and take one or two traces from each cluster. Without this step a thousand near-identical password-reset requests crowd out the single strange refund-in-another-currency case that actually costs you money.\n\nHamel Husain, who teaches evals to engineering teams, argues that the most valuable thing you can do is read real traces yourself and write down what went wrong in your own words. Do that for the first batch. Group your notes into categories such as wrong tool, ignored constraint, looping, or confident wrong answer. Those categories become the slice tags on your dataset, and they tell you what to measure before you've written a single grader.\n\n[How to Build an Eval Set for AI Agents Before You Trust One in Production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/)\n\nAir Canada was ordered to pay about C$812 in February 2024 after its chatbot invented a refund policy. This guide covers where to find test cases, how to score them with code before an LLM judge, and why each case should run three to five times. - [build an eval set for AI agents](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) - [AI agent test cases before production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/)\n\n## Label cheaply: grade the end state, not the wording\n\nLabelling is where these projects stall, because it feels like it needs an expert for every row. It mostly doesn't. Let a model draft the label: for each trace, have it propose what a correct outcome would have been and which criteria the run should meet. A person then accepts, edits or rejects the draft. Plan on a minute or two per trace as a rough working figure, which puts 150 traces at an afternoon or two for one reviewer.\n\nLabel the outcome, not the path. If the agent books a meeting, check that the calendar holds an event with the right attendee and time. If it resolves a support ticket, check the ticket status and the refund amount in the database. A reference transcript punishes an agent for taking a valid but different route, while an end-state check doesn't care how it got there. Use plain code assertions wherever you can, and keep binary pass or fail criteria for the rest, one failure mode per criterion.\n\nIf a model grades the outputs, calibrate it against people. Label 50 to 100 outputs by hand, run the judge on the same ones, and measure agreement with something like Cohen's kappa. A judge that disagrees with your reviewers on one case in four is producing a number you can't trust.\n\nHave two people label the first 30 cases independently. Where they disagree, the spec is vague. Fix the written criterion, not the reviewer.\n\nEach row should carry the same few fields so the set stays usable a year from now: a case ID, the input, the state snapshot, the expected end state, the graders to run, the slice tag, the source trace ID, the date added and who labelled it. The source trace ID matters most. It lets you go back to the original run when a case starts failing and nobody remembers why it was added.\n\n## LLM eval dataset size: what 100 cases can and can't tell you\n\nMost teams guess at LLM eval dataset size, so do the arithmetic instead. The uncertainty on a pass rate shrinks with the square root of your sample. At a true pass rate of 90%, 100 cases give a 95% confidence interval of about plus or minus 6 points. At 200 cases it's about 4 points, and at 400 it's about 3. A drop from 91% to 88% on a 100-case set is noise, and you shouldn't roll anything back over it.\n\nSlices make it worse. Split 200 cases into five categories and each one has 40, where a 50% pass rate carries a margin near 15 points. That's why a claim like the agent got worse at refunds is so hard to prove from a small set. Pick the two or three slices that matter most, usually the highest-revenue or highest-risk ones, and give each at least 100 cases. Let the rest stay small.\n\nComparing two versions on the same cases helps, because the same hard cases hit both and much of the noise cancels. Look at the cases that flipped, not only the averages. Ten cases that went from pass to fail are a list you can read in ten minutes.\n\nAgents add a second source of noise: they don't give the same answer twice. Run each case three to five times and record a pass rate per case. Anthropic's guidance on agent evals describes pass^k for this, the chance that all k runs succeed. An agent that succeeds 90% of the time per run passes five straight runs only about 59% of the time, since 0.9 to the fifth power is 0.59. If your users run the same task every day, that second number is the one they live with.\n\nFor a first version, 50 well-chosen cases beat 500 scraped ones. A sensible target is 100 to 200 total within a couple of months, with extra depth in your riskiest slices, and growth from there as incidents arrive.\n\n## Keep an AI agent regression test set that doesn't rot\n\nSplit the work into two tiers with different rules. The first is your AI agent regression test set, which is frozen. Every bug you fix adds a case, and no case leaves unless the feature it covers is removed. This set only grows, and a failure in it means something you already fixed has come back.\n\nThe second is the fresh set, a rolling sample of the last two to four weeks of traffic, sampled and labelled with the same method as above. It answers a different question: does the agent work on what people are asking right now? Throw it away and rebuild it every month instead of letting it accumulate. Old fresh cases that matter graduate into the regression set.\n\nWatch the input mix. Once a week, compare the share of traffic in each category against the share in your dataset. If a category that was 3% of requests in spring is 15% now, your set is underweighting it, and that's a trigger to add cases before anything breaks. The same goes for a new tool, a new model, a rewritten system prompt or a new product feature. Each one earns a refresh of the affected slices.\n\nRetire saturation, but carefully. When every version of your agent has passed a case for three months, it no longer separates good changes from bad ones. Move it to a small smoke set that runs on every commit, and keep it out of the number you report.\n\nVersion the dataset like code. Hash it, tag it, and record the hash alongside every eval run, otherwise a score change could come from the agent or from the cases and you won't know which. Also hold back about a quarter of the set and never tune prompts against it. Once you've iterated on a case until it passes, it measures your tuning, not your agent.\n\nA workable cadence looks like this: review 30 new traces every Friday, rebuild the fresh set each month, and make it a rule that no production incident closes until it has a case in the regression set. That last rule costs about ten minutes per incident, and it's the one that keeps the dataset pointed at your real users.\n\n**Also read:** [How to write an AI agent memory retention policy before it bites](https://startupfortune.com/how-to-write-an-ai-agent-memory-retention-policy-before-it-bites/) • [How to calibrate LLM as a judge before your eval dashboard lies to you](https://startupfortune.com/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you/) • [Build an LLM model deprecation migration plan before the cutoff week](https://startupfortune.com/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week/)\n\n*This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.*\n\n## Join the discussion\n\n[Open in the community →](https://startupfortune.com/community/)\n\nAlmost there. Sign in and your reply posts straight away.", "url": "https://wpnews.pro/news/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs", "canonical_source": "https://startupfortune.com/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs/", "published_at": "2026-11-21 19:26:58+00:00", "updated_at": "2026-11-21 19:41:45.129024+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "large-language-models"], "entities": ["Anthropic", "LangSmith", "Langfuse", "Braintrust", "Arize Phoenix", "OpenTelemetry"], "also_reported_by": [], "takeaway": "Build the golden set from logged production trajectories rather than hand-written prompts: sample failures plus a quarter to a third of successes, label end states, and run each case several times because 100 cases at a 90% pass rate carry about plus or minus 6 points of noise.", "recommendation": {"advice": "Harvest eval cases from production traces — failures plus a quarter to a third of successes — label the end state not the wording, and run each case multiple times since 100 cases at 90% pass rate carry roughly plus or minus 6 points of noise.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs", "markdown": "https://wpnews.pro/news/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs.md", "text": "https://wpnews.pro/news/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs.txt", "jsonld": "https://wpnews.pro/news/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs.jsonld"}}