Your PII redactor is not a secrets filter — and your LLM prompts are full of secrets

A developer built LeakProbe, a zero-dependency, vendor-neutral open-source tool that fires evasive secret and PII payloads at redaction layers to report what survives. Testing the popular OSS library redact-pii, LeakProbe found it caught 100% of classic PII but leaked 20 of 38 payloads (52.6%), including private keys, Stripe live keys, AWS secret access keys, OpenAI keys, and JWTs, while catching some credentials like Google API keys and Slack tokens. The developer argues the partial coverage is the real hazard, since a PII redactor is not a secrets filter and the failure lies in assuming one layer covers everything entering LLM prompts.

If you ship an LLM feature, you probably put a redaction layer in front of the model to strip sensitive data. Good. There's already solid, independent work measuring how well these tools catch PII — academic benchmarks like LeakageBench and REDACT, vendor comparisons on datasets like ai4privacy. If PII is your worry, that ground is covered. Here's what isn't measured: secrets. The API keys, tokens, connection strings and private keys that end up in prompts not because a user typed them, but because someone pasted a stack trace, an env dump, or a "why isn't this config working?" blob. Those are arguably worse to leak than a name — and nobody's independently testing whether your redactor stops them. So I built a small, zero-dependency, vendor-neutral tool — LeakProbe — that fires evasive secret and PII payloads at any redactor and reports what survives. Then I ran it against a real, popular OSS library: redact-pii . redact-pii is genuinely good at the job it advertises. Against my corpus it redacted 100% of the classic PII — emails, credit cards, IBANs, US SSNs — including spaced and lightly obfuscated variants that a naive regex misses. If you picked it to scrub PII, it earns its place. Here it gets interesting — the coverage is a patchwork : | Secret | Result | |---|---| | Google API key, Slack token, GitHub fine-grained PAT | Caught | | DB / Mongo connection strings password in the URL | Caught | | Private keys — RSA, EC, OpenSSH, even well-formed PEM | Leaked 100% | | Stripe live key, SendGrid key, AWS secret access key | Leaked | | AWS access key id, OpenAI key, classic GitHub PAT | Leaked | | JWT / Bearer token inside a JSON blob | Leaked | | Moroccan phone / national ID CIN | Leaked | This is sneakier than "PII tools are blind to secrets." redact-pii catches some credentials — Google, Slack, GitHub fine-grained tokens, even passwords inside connection strings. That partial coverage is exactly what lulls you into trusting it. Then your private keys, your Stripe live key, and your AWS secret access key sail straight through, and you cannot eyeball which half you got. Overall on my corpus: 20 of 38 payloads leaked, 52.6%. It's not that the library is broken — it's a PII tool, and a private key isn't PII. The failure is the deployment : assuming one redactor covers everything sensitive going into your prompts. Two takeaways: And these aren't hypothetical — the gaps mirror real, documented issues: streamed-output splits presidio 2312 https://github.com/data-privacy-stack/presidio/issues/2312 , silently-dropped locale entities presidio 2256 https://github.com/data-privacy-stack/presidio/issues/2256 , and secrets hiding in tool-call arguments litellm 41265 https://github.com/BerriAI/litellm/issues/41265 . LeakProbe ships a test for each. LeakProbe is open source, zero dependencies. Write a five-line adapter around whatever you use today and get a per-category leak report. git clone https://github.com/roshcompanylabs/leakprobe && cd leakprobe node src/cli.js run --target ./your-adapter.js --threshold 0 fail CI on any secret that leaks If you run it and something surprising leaks, I'd genuinely like to hear what.