I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree ( @dev_michael) . Instead I counted 204 of my own guards — and 89 % of them have never been asked to prove they can fail.
Michael wrote something that I couldn't put down: AI didn't make me a worse coder, it made me a worse reviewer. Here is the number, and it's worse than his thesis: of the 204 automated checks in my repositories that draw a conclusion, only 22 can prove they are able to fail. That's 11 %. The other 89 % have never once been shown a known-bad input. They are green. Whether they are green because everything is fine, or green because they are incapable of finding anything - I could not have told you last week. And I'm the person who wrote them.
First the definition, so you can reject it or reuse it.
A conclusion-bearing guard is any test that reads source code, config, or system state and asserts a claim about it. Not "does this function return 4" - but "no workflow downloads its cache over the network", "every page passes the same quarter filter", "this feature flag matches the deployed spec". The tests that stand in for a human reviewer.
A negative control is a probe that feeds that guard a known-bad input and asserts it gets rejected for the expected reason. Our convention marks them KONTROLLE:
in the test name.
Counting is mechanical: 204 guard files across three repositories, 22 with at least one control probe, 54 probes total. The counter is a proxy - marker-based, so unmarked controls and false-positive guard files put the true number at plus or minus a few points. The shape survives any correction: most of my reviewers have never been reviewed.
This isn't theoretical. All three of these happened to me in the last seven days, in production tooling.
The deploy gate that died of its own medicine. A pipeline step existed specifically to catch a silent failure mode - a missing tool falling back to an empty result. It called node -e
to parse a health response. The deploy runner has no Node. Six consecutive deployments failed with exit 127 - the check against missing tools failed on a missing tool, and nothing shipped for six hours. The step had been green in review because nobody had ever run it where it actually runs.
The harvester that threw away its own work. An autonomous job collected data from public repositories and judged each run by exit code. One run wrote seven perfectly good records, then hit a non-fatal warning and exited non-zero. The machine booked its own completed work as "failed, retry later" - because interrupted-with-partial-results had no representation, only success and failure. We caught it because the result file was sitting on disk right next to the exit code that denied its existence.
The pattern that matched the wrong 500. An error classifier looked for server errors with the pattern 50[024]
- anywhere in the output. It matched the "500" inside "4258 of 5000 quota points remaining" and classified a successful run as a server failure. Every field it read was real. It was answering a different question than the one asked.
Three different systems. One shape: the check watched a messenger - an exit code, a pattern, a status - while the artifact that mattered told a different story.
Here's where I think Michael's post lands harder than he says.
AI moved my job. I used to spend most of my day producing artifacts and a little of it verifying them. Now an agent produces most of the artifacts, and my job is verification. Which means my real codebase - the one my judgment actually ships through - is those 204 guards.
And that codebase is held to a standard I would reject in application code. No test coverage (11 %). No review of the reviewer. Green as the default state, silence booked as success.
When Michael says AI made him a worse reviewer, I'd sharpen it: AI promoted us all to reviewers, and none of us tested the reviewer. The model isn't the weak link. The unfalsifiable green checkmark is.
Everything above collapses into one sentence we now apply mechanically:
Judge the artifact, not the messenger.
Exit codes are messengers. Summaries are messengers. The agent's own "done" is a messenger. Green badges are messengers. The artifact is the diff, the file on disk, the served response body, the row in the database. When a messenger and an artifact disagree, the artifact is right - and a check that only ever reads messengers should be treated as unverified, however green it is.
The corollary for guards: a green zero is the most dangerous answer a check can give. "Found no violations" and "is incapable of finding violations" produce identical output. Only a negative control separates them.
This is the part you can use without believing me. Drop this in your repo root - it counts test files that read source or state, and how many carry a marked negative control (adjust the marker to your convention):
// count-controls.mjs — node count-controls.mjs
import { readFileSync, readdirSync, statSync } from "node:fs";
import { join } from "node:path";
const files = [];
(function walk(d) {
for (const n of readdirSync(d)) {
if (n === "node_modules" || n === ".git" || n === "dist") continue;
const p = join(d, n);
statSync(p).isDirectory() ? walk(p) : /\.test\.(t|j)sx?$/.test(n) && files.push(p);
}
})(".");
let guards = 0, withControl = 0, probes = 0;
for (const f of files) {
const t = readFileSync(f, "utf8");
if (!/readFileSync|readdirSync|execSync/.test(t)) continue; // "reads state" proxy
guards++;
const n = (t.match(/KONTROLLE|negative.control|can.?not.?find/gi) ?? []).length;
if (n) withControl++;
probes += n;
}
console.log(`${guards} conclusion-bearing guard files · ${withControl} with a negative control (${guards ? Math.round(100 * withControl / guards) : 0} %) · ${probes} probes`);
If your number is above 30 %, I'd genuinely like to know how you got there - that's the discussion I'm hoping for below.
Rule 2 of writing these posts is correcting yourself unprompted, so:
While building the feature this article's data comes from, my equivalence test failed by exactly 0.25 - and the bug was in my test, not the code: min-max spreading turns a column of zeros into a column of 0.5s and adds a constant. I had built a probe that answered a different question than the one asked, in the middle of measuring exactly that failure class.
And one push in that same hour went out with a red test - because npm test | grep
replaces the test's exit code with grep's. My pipeline read a messenger. The artifact - the failing test - sat right there.
The person telling you to test your reviewers failed to test his reviewer, twice, in one evening. That's not irony. That's the base rate, and it's why conventions beat discipline.
One developer, three repositories, one week - this is a case series, not a sample. The 11 % is marker-based and approximate. And I have not shown that raising falsifiability coverage improves outcomes downstream; I've shown that at 11 % I couldn't distinguish my working guards from my decorative ones. Whether the number that matters is 30 % or 80 %, I don't know yet - we're raising ours and measuring as we go.
There's also a fair objection: negative controls are themselves tests that can rot. True. But a control that rots fails loudly the next time the guard changes - that's the asymmetry that makes them worth writing.
So: what's your ratio? And more interesting - what's the greenest check in your pipeline that you now suspect has never been able to fail?
I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which earlier decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it — so nobody has to learn it twice.
Free tier, hosted in the EU: cachly.dev