If you build with AI every day, you know the pattern: the AI delivers, it runs, the tests pass, you nod. A few weeks in, nobody asks why it built things the way it did. With a UI you would notice, because you can see it. With logic in the background you see nothing. We caught ourselves doing it. This post is about the mechanism behind it, and what we changed.
Say a quote price comes together like this: material cost and setup time, plus a surcharge that depends on quantity. You write that down in one paragraph, attach a few old quotes as samples, and hand both to two people: a new colleague and an AI.
The colleague comes back an hour later with questions. What about a material that appears in none of the samples? At what quantity does the surcharge kick in, and does it jump or grow? And a third one you hadn't thought of.
The AI comes back a minute later with a calculation that runs and hits every sample to the cent.
"Looks good."
The AI had the same questions. It asked none of them and answered all of them, in the code, without a word.
A paragraph plus samples doesn't define one rule. It defines a family of rules: every rule that reproduces the sample results. A stepped surcharge and a linear one can agree on every quote you attached. So can a fallback that prices an unknown material like steel and one that throws an error.
The AI has to pick one member of that family. It picks one that fits the samples, and it doesn't mention that there was a choice.
Now look at where your tests come from. If they are built from the same samples, they check exactly what the AI already optimised for, and every rule in the family passes them. Green means the code fits the samples. Whether it implements the rule you meant, those tests can't say.
The difference shows up with the first input that isn't like the samples: the unusual material, the odd quantity. It shows up in production, not in the test run.
We saw this ourselves. When we asked the AI how exactly it calculates, its answer described a different rule from the one we meant. The tests had passed anyway, because the samples fit both rules.
The fix isn't less AI. It's a different start: the AI has to surface its decisions before it writes any code.
So before anything gets built, the AI answers in prose, with no code:
We answer, and it restates again. Round by round, until its restatement and our rule are the same rule. Only then does it build.
That changes two things downstream.
The tests come from the agreed rule, not just from the samples. Every question that came up in the rounds becomes a test case, so the suite finally holds inputs the samples never had. A passing test now confirms a rule both sides know.
And we still read the code, even when it runs. The quickest thing to look for is a constant or a special case that appears in the code and nowhere in the agreed rule. Each one is a decision nobody asked for.
Skip the rounds, and the decisions still get made. You've just handed them to someone who doesn't ask.
This report was written in German for decision-makers at knopfdruck.ai.