Most AI agents that fail in production never failed a single test, because nobody wrote one.
- A golden dataset is a fixed set of real, labeled transcripts pulled from actual usage, not ten conversations someone eyeballed before lunch
- LLM-as-judge scoring only counts once the judge model is checked against human raters, with most teams targeting a correlation above roughly 0.7 before trusting it
- Klarna's OpenAI-built support agent handled 2.3 million conversations and cut resolution time from 11 minutes to under 2, then the company had to rehire human agents in 2025 after customers hit its limits on nuanced cases
- Regression suites catch what a demo never will: a prompt fix that solves one complaint and silently breaks three others
- LangChain's 2026 State of AI Agents report found 57% of organizations now run agents in production, and quality, not cost, is the top barrier to shipping more
Everyone building an AI agent can demo it. Fewer can answer how do AI agent evals actually work once that same agent starts talking to real customers instead of a founder clicking through a familiar script, and that gap is exactly where most agentic products quietly break. The demo is scripted even when nobody scripted it on purpose. You ask the questions you already know it handles, in the order you know works, and it looks great. Production doesn't cooperate. Customers ask things sideways, chain three requests into one message, trigger a tool call that fails silently, and never bother telling anyone the answer was wrong. They just don't come back.
That's the actual failure mode killing agentic features right now, and it's rarely a dramatic outage. It's slow, silent decay: a retrieval step that starts pulling stale context after a vector store update, a tool call that times out and gets swallowed by a try and except block, a prompt tweak meant to fix one complaint that breaks five conversations nobody's watching. Nobody catches it, because nobody built anything that would catch it. Ask a founder how they know the agent still works after last week's prompt change, and the honest answer, most of the time, is that they tried it a couple of times themselves.
That's not testing. That's a vibe check.
Every real ai agent eval framework for startups rests on the same three pieces: a golden dataset of real, labeled cases, an automated judge that scores new runs against it, and a regression suite that reruns both automatically before anything ships. Here's how each one actually works, and where teams cut corners.
How To Evaluate AI Agents Before Production With a Real Eval Harness How to evaluate AI agents before production comes down to a golden dataset, trajectory-level scoring, and a CI gate that blocks bad deploys. Here's the actual mechanics behind the eval harnesses VCs now expect technical teams to show before they'll fund an agent product. - how to evaluate AI agents before production - building an eval harness for AI agents
A golden dataset is a fixed, reviewed set of real inputs paired with what a correct response actually looks like, along with a rubric for judging it. It gets pulled from actual production transcripts, or from a careful simulation of the messiest ones you expect, not from a handful of screenshots your team happened to like. Every case in it has been read and labeled by someone who understands the domain well enough to say this is right, or this is wrong, and here's why. According to Adaline's 2026 guide to LLM and agent evaluation, teams generally shouldn't trust an aggregate pass rate until the dataset clears somewhere around 500 labeled cases, because anything smaller is too noisy to tell a real regression from statistical wobble.
Hamel Husain, an ML consultant whose essay "Your AI Product Needs Evals" has become one of the most cited pieces on this exact problem, makes a sharper point than most teams want to hear: generic evaluation frameworks bolted onto your agent don't tell you much. The useful eval set is domain-specific, built from your own failure cases, scored against a rubric that matches what actually breaks for your product. A rubric for a support agent might check whether the response includes the correct refund policy and the customer's actual order number, not whether it phrases things elegantly. A rubric for a coding agent might check whether the generated function passes a real test suite, not whether it looks plausible. Exact string matching rarely works, because two correct answers can be worded completely differently. The rubric has to grade for the facts that matter and ignore the ones that don't.
Building this set is unglamorous work. Someone has to sit with fifty or a hundred real transcripts, mark each one pass or fail, and write down why. Most startups skip this step entirely, because it doesn't look like shipping.
LLM-as-judge scoring, and where it breaks down #
Once you have a golden set, you need a way to score new agent runs against it at a pace faster than a human can read every transcript. That's what LLM-as-judge does: a second model, usually a strong one, reads the agent's output alongside the rubric and the golden answer, and returns a verdict. Run this after every prompt change, every model swap, every retriever update, and you get real llm eval metrics before production traffic ever sees the change, instead of after a customer complains.
The catch is that the judge model isn't automatically trustworthy just because it's an LLM. It has to earn that trust the same way the agent does: by getting checked against real human raters on a subset of the golden set. Most teams building this kind of pipeline in 2026 are aiming for something like a Pearson correlation above roughly 0.7 between the judge's scores and a domain expert's verdicts before they'll let the judge run unsupervised. Below that, you're not measuring your agent. You're measuring how well two language models agree with each other, which tells you nothing about whether a customer got a correct answer.
Judges fail in specific, well-documented ways. They tend to favor longer answers over shorter correct ones, a pattern researchers call length bias. A model judging outputs from its own family sometimes rates them more generously than it rates a competitor's output on the identical task, a pattern known as self-preference. And judges drift: a rubric that scored strictly in January can quietly loosen up by June if nobody re-checks it against fresh human labels. None of this means LLM-as-judge is a bad idea. It means the judge needs its own upkeep and periodic recalibration, the same way any other part of the pipeline does.
Frankly, most teams skip the calibration step entirely. They wire up a judge prompt, get a number that looks reasonable, and never once check it against what a human would actually say. That number is worse than useless, because it creates false confidence.
How to Evaluate AI Agents Before You Ship Them to Real Users How to evaluate AI agents before launch comes down to building a real testing framework, not trusting a clean demo. This piece walks through the metrics, the eval set, and the failures at Chevrolet and McDonald's that show what happens when founders skip it. - how to evaluate AI agents - testing AI agents before production
Trajectory evaluation: grading the steps, not just the final answer #
Scoring only the last message an agent sends misses half the problem, because a multi-step agent can land on the right answer while taking a wrong path to get there. Picture a booking agent that's supposed to check a customer's loyalty tier before applying a discount. It skips that lookup, guesses the discount anyway, and happens to guess right. The final answer passes. The behavior underneath it is broken, and it will eventually guess wrong on a case your golden set never covered. This is why serious eval setups score the trajectory, the actual sequence of tool calls and intermediate steps, not just the output at the end.
A trajectory eval checks things a final-answer judge can't see: did the agent call the right tool, in a sane order, with the right arguments, and did it stop after getting a satisfactory result instead of calling the same API four more times. Some teams do this with a strict match against an expected call sequence. Others, especially where more than one valid path exists, use a second LLM judge that reads the full trace and grades whether the steps were reasonable, not just whether they matched a script exactly. Either approach costs more to build than a single output check, which is exactly why most startups skip it and only find out their agent's internal reasoning was broken after a customer hits the one path nobody tested.
Regression suites and how to test AI agents before you ship again #
A regression suite is the golden set plus the judge, run automatically every time something changes: a new prompt, a new model version, a new tool schema, a new retrieval index. The output is a score, compared against the last passing baseline, the same way a software team gates a deploy on unit tests. If the score drops, the change doesn't ship until someone understands why. This is ai agent regression testing in its plainest form, and it's the single piece most startups skip, because it only pays off after the second or third change, not the first.
The tooling for this has gotten real money behind it. Braintrust, which frames itself as an eval-first quality management layer rather than a general observability tool, closed an $80 million Series B in February 2026 led by ICONIQ, with Andreessen Horowitz and Greylock joining, plus the investor Elad Gil. LangSmith, built by LangChain, takes more of a trace-first approach, and its customer list includes Vercel and Notion, along with Coursera and Dropbox. There are lighter options too: whatbroke is an open-source command-line tool that simply diffs two agent trace files, before and after a change, so a small team can see exactly which conversations flipped from pass to fail without buying a platform at all. None of these tools do the actual work of writing the golden set or the rubric. They just make it possible to run that work continuously instead of once.
Klarna is the clearest real-world case of what happens when a team measures the wrong things instead of running this loop. In February 2024, chief executive Sebastian Siemiatkowski announced that the company's OpenAI-built customer service assistant had done the work of 700 human agents in its first month, handling 2.3 million conversations and cutting average resolution time from 11 minutes to under 2. Those are the metrics that get a headline. They say nothing about whether the answers were actually right for the harder cases. By May 2025, Klarna had reversed course and started rehiring human agents, after customers ran into an assistant that gave generic answers and struggled with nuanced, complicated requests. The company kept the bot handling roughly two-thirds of inquiries and added a guaranteed human option back on top, which is the honest version of the lesson: not less AI, but AI that gets checked against the cases it actually fails, not just the volume it processes.
That's exactly the gap a real eval suite closes. Resolution volume and speed are easy to measure and easy to celebrate. Whether the agent handled the one customer with a shipping dispute and a coupon code and a language barrier all at once is the harder question, and it's the one a golden dataset built from real edge cases is designed to answer before a customer has to find out the hard way.
So why does almost nobody do this before shipping? Speed, mostly. Writing a golden set and a rubric feels like a research project bolted onto a sprint, and founders racing to ship an agentic feature this quarter don't see the payoff until the second time they touch the prompt. According to LangChain's 2026 State of AI Agents report, 57% of organizations now have agents running in production, and quality, not infrastructure cost, ranks as the number one barrier to pushing further. That's not a coincidence. It's the direct cost of skipping the part of the process that has no demo of its own.
The fix isn't expensive. Pull fifty real transcripts from whatever channel your agent already runs in, mark each one pass or fail with a one-line reason, and rerun that same set every time you touch the prompt. That's an afternoon, not a quarter. If you can't do that in under five minutes right now, you don't actually know whether last week's change was an improvement or a regression. You're just hoping nobody notices before you do.
Also read: How Does an Earnout Work in an Acquisition, and Where Founders Get Shorted • How Do Drag-Along Rights Work in a Term Sheet, and Can They Force a Sale • Why AI Agent Approval Workflow Design Fails When Founders Rush It
This article is posted in AI News, check it out for more related stories.
Join the discussion #
Open in the community → Almost there. Sign in and your reply posts straight away.