My demo refused 96% of off-topic questions but only 39% of tricky ones. I know because of a test set

A developer built a labelled "golden set" test harness for a document Q&A demo and found it correctly refused 96.1% of entirely off-topic questions but only 39.4% of near-miss questions that sit close to a document without being answered by it. The same approach on a ticket-triage demo showed 90.2% accuracy if everything were automated versus 97.7% on only the confident 75.9% of cases. The developer argues that labelling roughly 100 real examples before a pilot is the cheapest acceptance test an AI feature can have.

Here are two numbers from my document Q&A demo. Ask it about something that is not in the knowledge base at all, and it refuses correctly 96.1% of the time. Ask it a question that sits close to a document but is not answered by it, and it refuses only 39% of the time. If I had tried the demo by hand, I would have asked a few questions, seen it say "not in the docs" a couple of times, and called it good. The 39% only showed up because there was a labelled test set. That set is the subject of this post: a golden set , the cheapest acceptance test an AI feature can have. What it is A golden set is a list of real inputs with the answer a human accepts for each one. Nothing more. It is not training data and it is not a benchmark you publish. It is the thing you run after every change to ask one question: did it get better or worse? Without it, every change is judged by whichever three examples someone happens to try. With it, "works pretty well" becomes a number you can argue about. How I would build one in an afternoon 1. Take real inputs. 100 is enough to start. Pull them from the actual process: tickets, invoices, questions people really ask. Not examples you invented to look nice. 2. Cover the ugly cases on purpose. Roughly: most of it typical, a quarter hard or ambiguous, and a handful that must be refused or sent to a human. The ugly part is where the demo above found its 39%. 3. Label it with the person who owns the process. The owner says what "correct" means. If two people disagree on a label, that disagreement is a finding, write it down. 4. Freeze it. Do not tune the system on these examples. Tune on a separate split and touch the golden set only to report. In my ticket-triage demo the threshold is chosen on a validation split and the test set is used once. 5. Write the acceptance rule before you run it. For example: "at least 97% correct on what is handled automatically, everything else goes to a person." Then the run is a pass or a fail, not an opinion. What it showed me in my own demos These are public demos on public datasets test data, not client results : - Ticket triage. If everything were automated, accuracy would be 90.2%. Handle only the confident 75.9% and accuracy on those is 97.7%. The golden set is what made the trade-off visible. - Document Q&A. Out-of-base questions are refused 96.1% of the time, but near-miss questions only 39.4%. The cited answers hit 93% precision against a 95% target. I reported the gap instead of hiding it, and it points to the next fix: a reranker or an LLM for exactly those near-miss cases. In both cases the number told me where to work next. A hand-tried demo would have told me nothing. Pitfalls - Leakage. If the system has seen the golden examples while being tuned, the score is fiction. - Too clean. A set made only of easy cases gives 99% and a false sense of safety. - Never refreshed. Real inputs drift. Add new failures to the set whenever production surfaces one. - One number for everything. Report per category. A fine average can hide a category that fails every time. Takeaways 1. A golden set is real inputs plus answers a human accepts. 2. Build it before the pilot, with the process owner. 3. Include the ugly cases deliberately. 4. Freeze it, and write the pass rule first. 5. Report per category, and add every new production failure to it. If you can only do one thing before your next AI pilot, label 100 examples. It is the cheapest hour of the project. Portfolio and contacts: @Kotenev01 bot https://t.me/Kotenev01 bot . I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: @kotenev01 en https://t.me/kotenev01 en