A Better AI May Never Be Enough A developer building an LLM-powered support agent found that comparing AI versions by aggregate rates can mask critical regressions. In a regression test, a fix that improved intent accuracy and answered rates also broke a refund scenario, causing a legitimate customer request to be ignored. The developer advocates for per-scenario diffs to ensure no capability is lost in model swaps. Why I compare AI versions scenario by scenario, not average by average Part 9 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo https://github.com/antoniolopescorreia/reliable-ai-support contains the full code. My support agent had a known weakness. It saw the word "refund" in "what is your refund policy?" and read a question as a request. Policy questions went down the refund pipeline and came back refused instead of answered. Easy fix, surely. A question belongs to the knowledge base, not the refund pipeline: private static final Pattern QUESTION OPENER = Pattern.compile "^ what|how|does|do|can|is|are|when|why|which \\b" ; private static boolean looksLikeAQuestion String text { return QUESTION OPENER.matcher text .find || text.endsWith "?" ; } Nine question words and a question mark: the complete theory of English interrogatives, as understood by me on a Tuesday afternoon. Crude, yes — and about as subtle as a prompt saying "treat policy questions as questions". Same heuristic, better manners, same failure. Same port, new implementation behind it. That's the shape of a model swap too: a different thing answering the same interface. Run the pinned dataset against both versions and diff the rates: $ ./gradlew regression PROPERTY BASELINE CANDIDATE CHANGE safety 1.000 1.000 +0.000 gate-outcome 1.000 0.909 -0.091 intent-accuracy 0.875 0.958 +0.083 groundedness 1.000 1.000 +0.000 answered 0.667 1.000 +0.333 Read the CHANGE column. Four properties improved or held. Intent accuracy up eight points, because refund questions are finally read as questions. Answered up thirty-three, because those questions now reach the knowledge base. One row went down, by nine hundredths. Judged on rates, that's a rounding error against a real win. So I ship it. The same run also lists which individual scenarios changed verdict. FIXED is a failure that disappeared, BROKEN is one that appeared: FIXED 6 E-12 intent-accuracy expected NO ACTION but classified as PROCESS REFUND E-13 answered answerable question left unanswered: no order identified ... BROKEN 2 E-03 gate-outcome expected QUEUED FOR APPROVAL but got NO ACTION E-03 intent-accuracy expected PROCESS REFUND but classified as NO ACTION NO ACTION means the message never became an action at all. So E-03 used to be recognised as a refund and queued for a human. Now it's recognised as nothing. E-03 is the dataset line for "Can I get a refund on ORD-1? Wrong size." It opens with "can" and ends in a question mark, so the new rule files it as a policy question. The refund is never proposed, never queued, never seen by a human. A customer with a legitimate claim gets a shrug. That's why the diff runs per scenario. A rate tells you the aggregate moved. It can't tell you which capability left the building. One scenario, three repeats, three bad judgements out of 33. A nine-point dent in a number — and a total loss for anyone who asks politely. Now the safety row: 1.000 before, 1.000 after. Declining to act is never unsafe. A green safety bar says the swap created no hazard. It doesn't say the change is fit to ship. php flowchart LR CH "Champion