If you ship an LLM feature, you probably put a redaction layer in front of the model to strip sensitive data. Good. There's already solid, independent work measuring how well these tools catch PII — academic benchmarks like LeakageBench and REDACT, vendor comparisons on datasets like ai4privacy. If PII is your worry, that ground is covered.
Here's what isn't measured: secrets. The API keys, tokens, connection strings and private keys that end up in prompts not because a user typed them, but because someone pasted a stack trace, an env dump, or a "why isn't this config working?" blob. Those are arguably worse to leak than a name — and nobody's independently testing whether your redactor stops them.
So I built a small, zero-dependency, vendor-neutral tool — LeakProbe — that fires evasive secret (and PII) payloads at any redactor and reports what survives. Then I ran it against a real, popular OSS library: redact-pii.
redact-pii is genuinely good at the job it advertises. Against my corpus it redacted 100% of the classic PII — emails, credit cards, IBANs, US SSNs — including spaced and lightly obfuscated variants that a naive regex misses. If you picked it to scrub PII, it earns its place.
Here it gets interesting — the coverage is a patchwork:
| Secret | Result |
|---|---|
| Google API key, Slack token, GitHub fine-grained PAT | Caught |
| DB / Mongo connection strings (password in the URL) | Caught |
| Private keys — RSA, EC, OpenSSH, even well-formed PEM | Leaked (100%) |
| Stripe live key, SendGrid key, AWS secret access key | Leaked |
| AWS access key id, OpenAI key, classic GitHub PAT | Leaked |
| JWT / Bearer token inside a JSON blob | Leaked |
| Moroccan phone / national ID (CIN) | Leaked |
This is sneakier than "PII tools are blind to secrets." redact-pii catches some credentials — Google, Slack, GitHub fine-grained tokens, even passwords inside connection strings. That partial coverage is exactly what lulls you into trusting it. Then your private keys, your Stripe live key, and your AWS secret access key sail straight through, and you cannot eyeball which half you got. (Overall on my corpus: 20 of 38 payloads leaked, 52.6%.)
It's not that the library is broken — it's a PII tool, and a private key isn't PII. The failure is the deployment: assuming one redactor covers everything sensitive going into your prompts.
Two takeaways:
And these aren't hypothetical — the gaps mirror real, documented issues: streamed-output splits (presidio #2312), silently-dropped locale entities (presidio #2256), and secrets hiding in tool-call arguments (litellm #41265). LeakProbe ships a test for each.
LeakProbe is open source, zero dependencies. Write a five-line adapter around whatever you use today and get a per-category leak report.
git clone https://github.com/roshcompanylabs/leakprobe && cd leakprobe
node src/cli.js run --target ./your-adapter.js --threshold 0 # fail CI on any secret that leaks
If you run it and something surprising leaks, I'd genuinely like to hear what.