chamnan is a Claude Code plugin that keeps engineering context in the repository, so an agent
stops rediscovering the same work every session. Part of that is a redactor: it scrubs what the
tool sends out, before it reaches a model backend.
It scored 98.3% recall on a labelled corpus. That number was true and it was hiding something.
A reader — Peter — made an argument on somebody else's
article: a pipeline can hit strong recall on straightforward payloads while systematically missing anything harder, and a blended recall number hides that completely.
So I split the figure by the retrieval problem each case poses. A credential can be named by an
assignment (API_KEY=...), positioned under a column header, described in surrounding prose, or
standing bare with nothing but its own characters to go on.
The column-header class had zero cases in it.
That is the class that leaked. A password under a CSV header — nombre,correo,contraseña —
escaped in every language the redactor claimed to handle, English included. The fix shipped. The
measurement that should have caught it had never sampled it once.
An empty class does not show up in a breakdown as a zero. It does not show up at all, which is
how it survived being looked at.
The corpus now derives its column cases from the redactor's own list of claimed languages, so a
language it claims is a language it is measured on. Sixteen languages, thirty-six cases.
All thirty-six pass — so the blended figure went up, 98.3% to 99.0%.
That is the same defect from the other side. A headline improved by testing more of what already
works flatters exactly as much as one that hides a weak class. So both numbers are published now:
99.0% overall, 93.8% in the weakest class, and a check fails if either goes missing from the
README or SECURITY.md.
Peter's other arguments turned into three more fixes:
check 4869 / 4869, and 1,466 of 1,466 index claims true, on Linux, macOS and Windows across two
Python versions.
git clone https://github.com/ArcticFox2029/chamnan
cd chamnan && python3 tools/verify_release.py
It runs the suite and the index-claim check on your machine and prints what happened. It refuses
to report a result when the totals line is missing, because a run that dies mid-way prints no
failure lines at all.
One honest limit: the recall corpus is synthetic, built to be hard to index, and is not part
of the repository — so you cannot reproduce those particular numbers from a clone. The suite and
the index claims you can.
MIT. Python standard library only. Nothing leaves the machine.