A blended recall number hid an entire class of leak

A developer behind chamnan, an MIT-licensed Claude Code plugin that redacts sensitive data before it reaches a model backend, split its 98.3% blended recall score by retrieval problem after a reader argued that aggregate figures can mask systematic failures on harder cases. The breakdown revealed the column-header class had zero test cases, and a password under a CSV header escaped in every language the redactor claimed to support; the corpus now derives column cases from the tool's own claimed languages, covering sixteen languages and thirty-six cases, all of which pass. Both figures are now published — 99.0% overall and 93.8% in the weakest class — with a check that fails if either is missing from the README or SECURITY.md.

chamnan is a Claude Code plugin that keeps engineering context in the repository, so an agent stops rediscovering the same work every session. Part of that is a redactor: it scrubs what the tool sends out, before it reaches a model backend. It scored 98.3% recall on a labelled corpus. That number was true and it was hiding something. A reader — Peter https://dev.to/peterbuildssecure — made an argument on somebody else's article: a pipeline can hit strong recall on straightforward payloads while systematically missing anything harder, and a blended recall number hides that completely. So I split the figure by the retrieval problem each case poses. A credential can be named by an assignment API KEY=... , positioned under a column header, described in surrounding prose, or standing bare with nothing but its own characters to go on. The column-header class had zero cases in it. That is the class that leaked. A password under a CSV header — nombre,correo,contraseña — escaped in every language the redactor claimed to handle, English included. The fix shipped. The measurement that should have caught it had never sampled it once. An empty class does not show up in a breakdown as a zero. It does not show up at all, which is how it survived being looked at. The corpus now derives its column cases from the redactor's own list of claimed languages, so a language it claims is a language it is measured on. Sixteen languages, thirty-six cases. All thirty-six pass — so the blended figure went up , 98.3% to 99.0%. That is the same defect from the other side. A headline improved by testing more of what already works flatters exactly as much as one that hides a weak class. So both numbers are published now: 99.0% overall, 93.8% in the weakest class , and a check fails if either goes missing from the README or SECURITY.md. Peter's other arguments turned into three more fixes: check 4869 / 4869 , and 1,466 of 1,466 index claims true, on Linux, macOS and Windows across two Python versions. git clone https://github.com/ArcticFox2029/chamnan cd chamnan && python3 tools/verify release.py It runs the suite and the index-claim check on your machine and prints what happened. It refuses to report a result when the totals line is missing, because a run that dies mid-way prints no failure lines at all. One honest limit: the recall corpus is synthetic, built to be hard to index, and is not part of the repository — so you cannot reproduce those particular numbers from a clone. The suite and the index claims you can. MIT. Python standard library only. Nothing leaves the machine.