{"slug": "a-detector-that-only-ever-says-clean-proves-nothing", "title": "A Detector That Only Ever Says \"Clean\" Proves Nothing", "summary": "A developer discovered that a code-scanning probe written for their coding agent failed to detect a script whose control label was in uppercase, returning a plausible but incorrect count. The incident highlights a broader problem: detectors that find nothing and detectors that cannot see produce identical output, and the developer proposes adding self-tests with positive and negative controls to all such tools.", "body_md": "A few days ago I asked my agent to count how many of my tooling scripts carry a self-test. It grepped and answered: 12 of 13.\n\nThe number was wrong. One script labels its control `НЕГАТИВНИЙ КОНТРОЛЬ`\n\n— uppercase — and the probe's regex was lowercase with no `-i`\n\nflag. The real answer was 13 of 13.\n\nA probe written to find blind detectors was blind. It returned a clean, specific, entirely plausible number, and nothing in its output hinted that it had missed anything. I caught it only because the total felt one short and I opened the file by hand.\n\nThat is the whole problem, and it took ninety seconds to demonstrate on myself.\n\nIf you work with a coding agent, you are accumulating detectors far faster than you notice. Not tests — *detectors*. Pre-commit hooks. Custom lint rules. Audit scripts. \"Check that no doc references a deleted file.\" \"Check that every rule in the project has an actual mechanism behind it.\" \"Check that the test count claimed in this commit message matches reality.\"\n\nThey cost one sentence to request, so you request them constantly. I have 29 in a single project. They run on every commit and they almost always print nothing, which is exactly what you want them to print.\n\nAnd there is the trap:\n\nA detector that found nothing and a detector that cannot see produce byte-identical output.\n\nSilence is the success state. Silence is also the total-failure state. You cannot tell them apart by looking — and the longer a detector stays quiet, the more you trust it, which is precisely backwards. A broken detector is silent *more* reliably than a working one.\n\nTest suites have a defence against this. [Mutation testing](https://en.wikipedia.org/wiki/Mutation_testing) perturbs your production code, re-runs the suite, and reports any mutant that survived — a change nothing caught. It exists because you can reach 100% line coverage with tests that assert nothing at all.\n\nBut mutation testing points at *suites*, in CI, over application code. Nobody mutation-tests the 200-line script their agent wrote on Tuesday to check something about their docs. And the guardrail tooling that grew up around LLM coding — pre-call and post-call interceptors — is built to [stop the model from doing something dangerous](https://www.arthur.ai/blog/best-practices-for-building-agents-guardrails), not to prove that a bespoke checker can still see.\n\nSo the fastest-growing category of quality machinery in your repo is the one with no soundness check at all.\n\nEvery assay ships with controls. A [negative control](https://www.nist.gov/glossary-term/36651) is the assay run with everything *except* the sample: no signal is expected, and if a signal shows up, the run is contaminated and its results are void. A positive control is the mirror — a known-present sample that **must** produce a signal. If it doesn't, the instrument is dead, and every clean reading it gave you today means nothing.\n\nThe translation to software is direct. Give every detector a `--self-test`\n\nflag. Behind it, paired controls:\n\nRun the controls before trusting the report. If any control fails, the tool does not print a verdict at all. It prints that it is unsound.\n\nHere is the real thing, trimmed, from a tool that audits whether every rule in my project has an enforcement mechanism behind it:\n\n```\n// A detector that only ever says \"clean\" proves nothing.\nfunction selfTest(S) {\n  const ok = [], bad = [];\n  const t = (name, got, want) =>\n    (String(got) === String(want) ? ok : bad).push(`${name}: got ${got}, want ${want}`);\n\n  // Can it still read its inputs at all?\n  t(\"sources · routing table found\",  S.table.length > 500,    \"true\");\n  t(\"sources · test classes parsed\",  S.testClasses.size > 50, \"true\");\n\n  // Positive: a rule that IS in the table must resolve.\n  t(\"positive · slug fully in table\",\n      slugHit(norm(S.table), \"smart-design\"),            \"full\");\n\n  // Vacuum: a rule that exists nowhere must resolve to nothing.\n  t(\"negative · absent slug\",\n      slugHit(norm(S.table), \"zzz-nonexistent-rule\"),    \"null\");\n\n  // Vacuum: an invented filename must not be accepted as a witness.\n  t(\"rules-witness · vacuum — invented file is not\",\n      S.rulesTests.has(\"no-such-file.test.js\"),          \"false\");\n\n  console.log(bad.length\n    ? \"❌ DETECTOR UNSOUND — do not trust its report\"\n    : \"✅ controls pass — detector may be trusted for this run\");\n  return bad.length === 0;\n}\n```\n\nLook at the last line. The tool never claims the codebase is clean. It claims that *for this run, its own verdict is worth reading*. Those are different statements, and keeping them apart is most of the value.\n\nControls go blind too. This is where it stops being trivial.\n\nOne of my audits reads a corpus of documentation and reports **orphans** — guards that exist in code but that no document explains. It needed a positive control: some token guaranteed to be present, so that a `false`\n\nwould mean \"the reader is broken,\" not \"this guard is undocumented.\"\n\nThe first control I wrote picked a guard that, as it turned out, genuinely *was* undocumented. So it returned `false`\n\n. And `false`\n\nthere is indistinguishable from a completely broken corpus reader. A control designed to prove the reader worked would have quietly certified a reader that had stopped reading.\n\nThe comment in that file now reads:\n\n```\n// Positive control for the orphan check: EtalonChainGuardTest is the best-explained\n// guard in the project (6 memory files + 5 docs). If the corpus is being read at all,\n// this token is present — so a `false` here means the reader is broken, not that the\n// guard is an orphan. (An earlier control used a guard that is in fact undocumented,\n// which would have made a broken reader look correct.)\n```\n\nThe rule I took from it: **anchor a positive control to the most redundantly-present fact you have**, never to a convenient example. If the anchor is marginal, its absence is ambiguous — and an ambiguous control is not a control.\n\nI found it the day after publishing these tools as [a public repository](https://github.com/VolodymyrKubiria/negative-control), in the one script that grades all the others.\n\nThat harness has a mutation mode: it blinds a hook on purpose and requires every positive case to go silent. A green run means the controls are capable of failing. I ran it and got a flawless score for two of three hooks — and the score meant nothing. The mutation expression replaced the *first* line of a multi-line pipeline and orphaned its continuations, so the mutant no longer parsed. Bash never started the hook. Every case \"went silent\" for a reason that has nothing to do with blindness, and the report concluded: *every EXPECT went silent — these controls can actually fail.*\n\nThe harness already carried a control for this family. It refuses a mutation that changes nothing, on the grounds that such a mutation looks identical to one that worked. That control ran, and passed: the file *had* changed.\n\n**Changed and still executable are two different questions**, and only the second one makes the silence mean what the report says it means. One line separates them — run the mutant through `bash -n`\n\nbefore grading anything. It is now control ⑨, with a paired control ⑨b, because a parse check that rejected every mutant would have passed ⑨ alone.\n\nI keep relearning the same shape. The detector is a hypothesis about the subject; the control is a hypothesis about the detector; and the thing that runs the controls is a hypothesis nobody had written down.\n\nMinimum viable version, any language:\n\n`--self-test`\n\nflag that runs before anything else and exits non-zero on failure.Step 6 is where it compounds. My detectors' control lists now read like a diary of every way each one has previously been wrong. That list is the actual asset; the detector is just the thing it's attached to.\n\nIf you would rather start from working code than from a list, the same practice is packaged in [ negative-control](https://github.com/VolodymyrKubiria/negative-control) — MIT, three bash hooks plus two harness scripts and one\n\n`.mjs`\n\n, no install. The snippets above are from a private project; the repository holds generalized copies, each with its own controls. `bash scripts/probe.sh --all`\n\nruns them, and `--mutate`\n\nblinds a guard on purpose so you can watch its controls catch it.It is not a replacement for tests. It is not mutation testing — mutation perturbs the *subject* to grade the *checker*, which is stronger and considerably more expensive. Self-tests with controls perturb nothing. They just refuse to let a checker report until it has shown, on fixed fixtures, that it can still tell signal from noise.\n\nNor does it escape the regress. Who controls the controls? Nothing does. The controls are hand-written fixtures and they can rot right alongside the code. What the practice buys is a floor, not certainty: a detector without controls can be blind from birth and never say so, while a detector with controls has to survive a named list of things it must catch and must ignore — and when one of those breaks, it breaks loudly instead of printing a reassuring nothing.\n\nIn a solo-built production Android app — 73,411 lines of Kotlin, 1,683 unit tests, all measured 2026-08-16:\n\n`--self-test`\n\nFourteen tools have none. They are the ones I trust least, which is the correct amount.\n\nThe opening anecdote counted 13 of 13. That was a few days earlier, and two more tools have grown controls since — which is precisely why every number here carries a date and none of them is worth repeating without one.\n\n**Limits, stated plainly.** These counts come from grepping my own repository — the same method that was wrong in the opening anecdote before I checked it by hand. I haven't surveyed how common the practice is elsewhere; my impression that it's rare comes from searching, not from reading other people's code, and absence from a search result is not absence from the world. And a passing control suite proves only that the detector could see *at the moment the controls ran*, on the fixtures it was handed. Nothing beyond that.\n\nWhich is still a great deal more than green.\n\n*If you have a checker in your pipeline that has been quietly green for six months, run the experiment: hand it something it should catch. Five minutes, and the result is always interesting.*", "url": "https://wpnews.pro/news/a-detector-that-only-ever-says-clean-proves-nothing", "canonical_source": "https://dev.to/volodymyrkubiria/a-detector-that-only-ever-says-clean-proves-nothing-mii", "published_at": "2026-08-16 17:09:09+00:00", "updated_at": "2026-08-16 17:42:15.657980+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-safety"], "entities": ["Arthur AI", "NIST"], "alternates": {"html": "https://wpnews.pro/news/a-detector-that-only-ever-says-clean-proves-nothing", "markdown": "https://wpnews.pro/news/a-detector-that-only-ever-says-clean-proves-nothing.md", "text": "https://wpnews.pro/news/a-detector-that-only-ever-says-clean-proves-nothing.txt", "jsonld": "https://wpnews.pro/news/a-detector-that-only-ever-says-clean-proves-nothing.jsonld"}}