At Fin, we deploy to production around 200 times a day; shipping is our heartbeat. To do so, we rely a lot on testing. But you can’t test an AI Agent the way you test normal software, for a few reasons.
First, AI is probabilistic: ask the same question twice and you can get two different answers.
Second, AI is unpredictable: a small prompt or input change can have a non-linear impact on behavior.
Third, the unit of output is really a whole dynamic conversation: changes can affect what an Agent says, when it asks a clarifying question, when it calls a tool, and so on.
Over three years and thousands of experiments on our own AI Agent Fin, we developed a rigorous scientific evaluation process to deal with this. Every change, from a one-line prompt tweak to swapping the underlying model, has to earn its way to production through backtests (replaying past conversations offline), then an A/B test with real customers, then a monitored rollout.1
From my experience, if you are serious about AI, you have to be serious about evals.
Why you can’t just ship #
Here is a real example we faced in 2024 of the difficulties of AI evaluation: Fin used to answer questions based on a summary of the customer’s issue, not the full back-and-forth of the conversation. Giving Fin the whole chat history was an obvious improvement once LLMs got more powerful. We tried such a change, but received a surprising alert from our hallucination judge (an LLM used to judge another LLM), which flags answers ungrounded in the sources. Hallucinations had increased significantly!
With more context, Fin tried to be more like a human support rep and started promising things it didn’t have the capability to do at the time, like telling customers it had already processed their refund. (Fin can now do this with Procedures.)
That forced our hand and we had to block that release. It took about two months of further prompt changes, backtests, and reviews to ship a version with a higher resolution rate and fewer hallucinations than we started with. Without that judge, we would have shipped a worse Agent.
Backtests: replaying the past #
“Past performance is no guarantee of future results.” – Finance saying
A backtest answers a simple question: if this change had been live last week, what would the Agent have said or done differently?
We take a few thousand real past conversations and replay them offline through two versions of the system, current production and the candidate, using the exact same inputs both times. The only difference between the two arms is the change itself, so any difference in output was caused by it.
That counterfactual is what makes backtests so useful: you can read the old answer and the new answer for the exact same conversation side by side. A/B tests can’t do that, since each group sees different conversations, which also makes backtests far more sample efficient. And no customer ever sees a backtest, so a terrible idea costs you an afternoon instead of an exec-level escalation (trust me, I’ve been there).
Look at the data!
“Manual inspection of data has probably the highest value-to-prestige ratio of any activity in machine learning.”– Greg Brockman, President of OpenAI
How do you score a few thousand replayed conversations? Simply start by reading them. Go through a sample of the actual conversations. No need to be fancy: a spreadsheet is fine. This is where you catch the failure modes you didn’t think to check for, and where the next automated checks come from. Also, it develops your intuition and understanding of the system.
Manual analysis stops scaling somewhere in the hundreds. After that, it is the job of LLM judges: give one or more LLMs the conversation, the answer, and a rubric, and ask for a verdict. For Fin, we want to know: is the answer grounded in the sources? Does it follow the customer’s configured guidance? Did it escalate when it should have? Judges can grade every conversation from a backtest in less than an hour.
Judges are LLMs grading LLMs[2](#footnote-2), so watch out:
Use an ensemble. One judge from one model provider has that provider’s blind spots. Our hallucination checker combines judges from different model families and sometimes calibrates the combination against human labels. Use disagreement between judges as a proxy for problematic cases (either the answer is marginal or the judges are confused).
Expect noise. Run the same judge twice on the same answer and some verdicts flip. Individual verdicts are unreliable so rely on averages and paired comparisons, but manually investigate outstanding cases (error analysis).
Who watches the watchmen? Periodically, humans should label a sample and measure the judges against those labels. One such exercise showed our guidance judges were too harsh: Fin followed the spirit of the rules much better than the judges implied. Without the check, we would have kept blocking good changes on the word of a miscalibrated critic.
A/B tests: the gold standard #
“Any figure that looks interesting or different is usually wrong.”– Twyman’s law
Backtests kill bad ideas cheaply, but they can’t tell you the things that matter most: how real end users will react to the change and its impact on business metrics. An A/B test is the gold standard for decision making: part of live traffic gets the change, the rest stays on the current system, and you check the difference. A/B tests provide a causal estimate of impact: traffic is split at random, so any difference between the arms comes from the change itself (or chance, which the statistics below let you bound).
The metric that anchors our A/B tests is resolution rate: the share of conversations Fin handles that it actually resolves, either confirmed by the end user or inferred because no human was needed afterwards.3
Gains compound over time. When Fin launched in early 2023, its resolution rate was around 30%, which we thought was a strong number already. Today, it’s 76%, and Fin’s CSAT climbed from 57% to 75% along the way, showing we didn’t just make Fin more deflective. No single release explains that: it was a long tail of small improvements, each one earning its way through this process, all confirmed with A/B tests.
One number cannot capture an AI Agent, though, so we read every experiment against a wider scorecard: hard resolutions (the end user explicitly said it helped), negative feedback, customer satisfaction (CSAT and our own CX score), latency, costs, hallucination rate, guidance following, and even answer length and formatting. Changes that improve everything at once are rare. Most ship decisions are trade-offs: you can raise resolution rate by making it harder to reach a human, but satisfaction drops. You can raise it by ignoring the escalation rules a customer configured, but their trust drops. You can make Fin smarter by letting it think longer, but every answer gets slower and more expensive.
Statistical savviness pays off here, but in practice it comes down to a short list:
Calculate your sample size upfront. A power analysis tells you how long the test must run to detect the effect you care about, and stops you from peeking early and calling a noisy blip a win.
Declare one primary metric before you start. Picking the best-looking metric after the results are in is just p-hacking.
Consider seasonality. Monday traffic is not Saturday traffic, so run tests in whole weeks (or longer for weaker effects).
Quantify your uncertainty with p-values and confidence intervals, with a stricter bar than the usual 5%, because when you run experiments continuously a lax bar guarantees false discoveries.
Watch out for slicing and too many metrics. With enough segments, one of them is always statistically significant by pure chance.
We also ramp rather than flip: a change might run dark at 0% first, executing invisibly so we can check errors and latency, then 1%, then more, with health checks at each step.
The final decision is never a binary readout of the primary metric. We have two standing rules: never knowingly ship a resolution rate regression, and never knowingly increase hallucinations. Around them sits a holistic judgement call: what the judges say, what the manual reads showed, what it costs, and how fast it is. We have held back changes that clearly won on resolution rate because judge panels showed the surrounding conversations had issues like worse handoffs to human reps. Weighing all of that takes judgment and taste, and those only come from running the loop end to end, hundreds of times, over years.
After the release #
The system you evaluated is not the system you’ll be running in three months: models get updated, content changes, end users ask about things that didn’t exist when you ran your backtest. So the work never really stops (at least we have job security in the age of AI!). In practice, we monitor Fin across the board in two different layers:
The first layer is classic production monitoring, no different from any serious software system: dashboards and alerts on the four golden signals of SRE (latency, traffic, errors, saturation), with our business metrics on top, resolution rate first among them. The second layer is AI-specific: judges keep scoring samples of live conversations, and humans keep reading them.
When monitoring surfaces a new failure mode, it becomes a case in the next backtest, and the loop closes. Post-release monitoring deserves a post of its own from our AI Infra engineers, so I’ll stop here.
Conclusion #
Evaluating an AI Agent is a multi-stage process: read the data yourself, backtest offline with deterministic checks and LLM judges, A/B test with real customers on a business metric, and keep monitoring after the release. My takeaways after years of running this loop on Fin:
Serious AI requires serious evals.
Look at the data!
A/B tests are the gold standard, so brush off your stats.
Treat production AI as software engineering plus ML, not either alone.
For most of Fin’s life this process was internal tooling, judges, and dashboards. But any company deploying an AI Agent faces the same problem we do, they’re changing the behaviour of a probabilistic system in front of real customers. For those configuring Fin, we decided to bake the same discipline our ML team uses into the product – allowing support teams to test offline, experiment safely, and monitor everything. We call it eval-driven delivery. However you build it, rigorous evaluation is how you make an AI system better, release after release, without breaking what already works.
1 Not to be confused with “eval-driven development,” a test-driven-development analogy where you write evals before building the feature. That tends to mean writing evaluations for problems you imagined instead of problems you have. This post is about the release path: what evidence a change must produce before it reaches more of your customers.
2 AI hacker koan:
After a lecture on evaluating large language models, a researcher was approached by an engineer.
“Your claim that we can evaluate an LLM by comparing its outputs against human judgments sounds very convincing,” said the engineer, “but it doesn’t scale. I have a better approach.”
“And what is that?” asked the researcher.
“We use a stronger LLM as a judge.”
The researcher d.
“But how do you know that the LLM judge is a good judge?”
“That’s easy. We evaluate it with another, stronger LLM.”
“And how do you evaluate that LLM?”
The engineer smiled triumphantly.
“You don’t understand. It’s LLM judges all the way down.”
3 We call the first kind a hard resolution and the second a soft resolution. We prize hard resolutions a lot more and try never to reduce them.