Every retrieval-augmented chatbot demo looks the same: paste in a folder of PDFs, ask a question, get a plausible answer, applause. Then it meets real users, who ask real questions, and the plausible answers start being wrong in ways nobody planned for.
In one line: A regulated financial client's document chatbot shipped only after a weighted 21-case evaluation, nine rounds of structured QA and a daily accuracy report the client ran themselves.
I spent most of this year taking one such chatbot from a demo to a client beta for a wealth-management firm in a regulated market. The model mattered less than I expected. The evaluation mattered more than anything else. This is the evaluation.
Before writing a single test case I listed the ways the bot could be wrong. Three families covered almost everything:
Each family got seven cases, so 21 in total. Not more. A test set you can run by hand in an afternoon gets run. A test set of 300 gets automated badly or ignored.
| Family | What it checks | A typical case | Weight |
|---|---|---|---|
| Intent | The bot answers the question that was asked, not a nearby one | A four-word question in the client's internal jargon | 1 to 2 |
| Retrieval | The right passage reaches the model | A figure that lives only inside a chart on a slide | 1 to 2 |
| Semantic drift | The answer says only what the passage says | A number the passage gives to two decimals, which the bot must not round | 2 |
| Scope, inside each family | The bot declines or redirects what it must not answer | "Which of these two funds should my mother buy" | 3 |
Scope cases are spread across the three families rather than kept as a fourth, because a scope failure usually looks like an intent or retrieval failure until you read the answer. Every weight-3 case is a scope case.
Each case carries a weight of 1, 2 or 3, set by what a wrong answer would cost the client, not by how hard the case is.
A round's score is the weighted pass rate: the sum of the weights of the cases that passed, divided by the sum of all weights. A case passes only if a reviewer reads the answer and agrees it is correct, cited and inside scope; a near miss is a fail.
Two rules sit on top of the score:
The weight-3 cases were all about scope: questions the bot must decline or redirect, phrased in ways that make declining hard. "Which of these two funds should my mother buy" is a weight-3 case. The correct answer is a polite refusal plus a pointer to a human, and it has to be the correct answer every time.
We ran the 21 cases as a formal round after every meaningful change, nine rounds in total before beta. A sample of what each round caught:
Rounds six to nine were smaller: citation formatting, a date-parsing bug, and a case where the correct answer was "the document does not say", which the bot found very hard to produce.
The release criteria lived on one page, and every round ended by checking it:
The last criterion is the one people skip. The beta did not open until the client could measure it without us.
The most useful thing we did was let the client's operations team score the bot daily, in their own spreadsheet, against their own real questions. Every incorrect answer got one of three labels:
A typical day in the beta: 79 queries, 87 percent correct, 10 incorrect, of which 7 were bot issues, 2 were user-understanding cases, and 1 was a duplicate. The trend over the first weeks moved between 82 and 91 percent. Two things came out of this that no internal test could have given us.
First, the split told us where to spend effort. When document issues dominate, the fix is content governance, not engineering. When user-understanding dominates, the fix is a clarifying-question behaviour, not a better model.
Second, the client stopped asking "is it accurate?" and started asking "what is today's number and why did it move?". That is the conversation you want.
The prototype was a React front end talking to a Flask script with a FAISS index. For beta it became a containerised FastAPI service with Celery workers, Redis, a Milvus vector store and PostgreSQL, behind Traefik, with images built in CI and pulled from a registry. None of that improved accuracy. It made the evaluation repeatable, which is what improved accuracy.
Lessons, in the order we learned them:
The bot is still in beta. The number still moves. But every move now has a reason attached to it, and that is the difference between a demo and a product.
Originally published at kalpakkale.com, where I write Dispatches, field notes on putting AI to work.