{"slug": "a-blended-recall-number-hid-an-entire-class-of-leak", "title": "A blended recall number hid an entire class of leak", "summary": "A developer behind chamnan, an MIT-licensed Claude Code plugin that redacts sensitive data before it reaches a model backend, split its 98.3% blended recall score by retrieval problem after a reader argued that aggregate figures can mask systematic failures on harder cases. The breakdown revealed the column-header class had zero test cases, and a password under a CSV header escaped in every language the redactor claimed to support; the corpus now derives column cases from the tool's own claimed languages, covering sixteen languages and thirty-six cases, all of which pass. Both figures are now published — 99.0% overall and 93.8% in the weakest class — with a check that fails if either is missing from the README or SECURITY.md.", "body_md": "chamnan is a Claude Code plugin that keeps engineering context in the repository, so an agent\n\nstops rediscovering the same work every session. Part of that is a redactor: it scrubs what the\n\ntool sends out, before it reaches a model backend.\n\nIt scored 98.3% recall on a labelled corpus. That number was true and it was hiding something.\n\nA reader — [Peter](https://dev.to/peterbuildssecure) — made an argument on somebody else's\n\narticle: *a pipeline can hit strong recall on straightforward payloads while systematically missing anything harder, and a blended recall number hides that completely.*\n\nSo I split the figure by the retrieval problem each case poses. A credential can be named by an\n\nassignment (`API_KEY=...`), positioned under a column header, described in surrounding prose, or\n\nstanding bare with nothing but its own characters to go on.\n\nThe column-header class had **zero cases in it.**\n\nThat is the class that leaked. A password under a CSV header — `nombre,correo,contraseña` —\n\nescaped in every language the redactor claimed to handle, English included. The fix shipped. The\n\nmeasurement that should have caught it had never sampled it once.\n\nAn empty class does not show up in a breakdown as a zero. It does not show up at all, which is\n\nhow it survived being looked at.\n\nThe corpus now derives its column cases from the redactor's own list of claimed languages, so a\n\nlanguage it claims is a language it is measured on. Sixteen languages, thirty-six cases.\n\nAll thirty-six pass — so the blended figure went **up**, 98.3% to 99.0%.\n\nThat is the same defect from the other side. A headline improved by testing more of what already\n\nworks flatters exactly as much as one that hides a weak class. So both numbers are published now:\n\n**99.0% overall, 93.8% in the weakest class**, and a check fails if either goes missing from the\n\nREADME or SECURITY.md.\n\nPeter's other arguments turned into three more fixes:\n\n`check 4869 / 4869`, and 1,466 of 1,466 index claims true, on Linux, macOS and Windows across two\n\nPython versions.\n\n```\ngit clone https://github.com/ArcticFox2029/chamnan\ncd chamnan && python3 tools/verify_release.py\n```\n\nIt runs the suite and the index-claim check on your machine and prints what happened. It refuses\n\nto report a result when the totals line is missing, because a run that dies mid-way prints no\n\nfailure lines at all.\n\n**One honest limit:** the recall corpus is synthetic, built to be hard to index, and is not part\n\nof the repository — so you cannot reproduce those particular numbers from a clone. The suite and\n\nthe index claims you can.\n\nMIT. Python standard library only. Nothing leaves the machine.", "url": "https://wpnews.pro/news/a-blended-recall-number-hid-an-entire-class-of-leak", "canonical_source": "https://dev.to/arcticfoxz_255bbe944bc015/a-blended-recall-number-hid-an-entire-class-of-leak-ni6", "published_at": "2026-09-11 06:00:13+00:00", "updated_at": "2026-09-11 06:55:56.103658+00:00", "lang": "en", "topics": ["ai-tools", "ai-safety", "developer-tools", "ai-agents"], "entities": ["chamnan", "Claude Code", "Peter", "ArcticFox2029", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/a-blended-recall-number-hid-an-entire-class-of-leak", "markdown": "https://wpnews.pro/news/a-blended-recall-number-hid-an-entire-class-of-leak.md", "text": "https://wpnews.pro/news/a-blended-recall-number-hid-an-entire-class-of-leak.txt", "jsonld": "https://wpnews.pro/news/a-blended-recall-number-hid-an-entire-class-of-leak.jsonld"}}