# Your PII redactor is not a secrets filter — and your LLM prompts are full of secrets

> Source: <https://dev.to/roshcompanylabs/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of-secrets-3ig8>
> Published: 2026-10-23 12:33:03+00:00

If you ship an LLM feature, you probably put a redaction layer in front of the model to strip sensitive data. Good. There's already solid, independent work measuring how well these tools catch **PII** — academic benchmarks like LeakageBench and REDACT, vendor comparisons on datasets like ai4privacy. If PII is your worry, that ground is covered.

Here's what *isn't* measured: **secrets.** The API keys, tokens, connection strings and private keys that end up in prompts not because a user typed them, but because someone pasted a stack trace, an env dump, or a "why isn't this config working?" blob. Those are arguably worse to leak than a name — and nobody's independently testing whether your redactor stops them.

So I built a small, zero-dependency, vendor-neutral tool — **LeakProbe** — that fires evasive secret (and PII) payloads at any redactor and reports what survives. Then I ran it against a real, popular OSS library: `redact-pii`.

`redact-pii` is genuinely good at the job it advertises. Against my corpus it redacted **100% of the classic PII** — emails, credit cards, IBANs, US SSNs — including spaced and lightly obfuscated variants that a naive regex misses. If you picked it to scrub PII, it earns its place.

Here it gets interesting — the coverage is a **patchwork**:

| Secret | Result | 
|---|---|
| Google API key, Slack token, GitHub fine-grained PAT | Caught | 
| DB / Mongo connection strings (password in the URL) | Caught | 
| **Private keys — RSA, EC, OpenSSH, even well-formed PEM** | **Leaked (100%)** | 
| **Stripe live key, SendGrid key, AWS secret access key** | **Leaked** | 
| **AWS access key id, OpenAI key, classic GitHub PAT** | **Leaked** | 
| **JWT / Bearer token inside a JSON blob** | **Leaked** | 
| Moroccan phone / national ID (CIN) | **Leaked** | 

This is sneakier than "PII tools are blind to secrets." `redact-pii` catches *some* credentials — Google, Slack, GitHub fine-grained tokens, even passwords inside connection strings. That partial coverage is exactly what lulls you into trusting it. Then your **private keys, your Stripe live key, and your AWS secret access key** sail straight through, and you cannot eyeball which half you got. (Overall on my corpus: 20 of 38 payloads leaked, 52.6%.)

It's not that the library is broken — it's a *PII* tool, and a private key isn't PII. The failure is the **deployment**: assuming one redactor covers everything sensitive going into your prompts.

Two takeaways:

And these aren't hypothetical — the gaps mirror real, documented issues: streamed-output splits ([presidio #2312](https://github.com/data-privacy-stack/presidio/issues/2312)), silently-dropped locale entities ([presidio #2256](https://github.com/data-privacy-stack/presidio/issues/2256)), and secrets hiding in tool-call arguments ([litellm #41265](https://github.com/BerriAI/litellm/issues/41265)). LeakProbe ships a test for each.

LeakProbe is open source, zero dependencies. Write a five-line adapter around whatever you use today and get a per-category leak report.

```
git clone https://github.com/roshcompanylabs/leakprobe && cd leakprobe
node src/cli.js run --target ./your-adapter.js --threshold 0   # fail CI on any secret that leaks
```

If you run it and something surprising leaks, I'd genuinely like to hear what.