{"slug": "your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of", "title": "Your PII redactor is not a secrets filter — and your LLM prompts are full of secrets", "summary": "A developer built LeakProbe, a zero-dependency, vendor-neutral open-source tool that fires evasive secret and PII payloads at redaction layers to report what survives. Testing the popular OSS library redact-pii, LeakProbe found it caught 100% of classic PII but leaked 20 of 38 payloads (52.6%), including private keys, Stripe live keys, AWS secret access keys, OpenAI keys, and JWTs, while catching some credentials like Google API keys and Slack tokens. The developer argues the partial coverage is the real hazard, since a PII redactor is not a secrets filter and the failure lies in assuming one layer covers everything entering LLM prompts.", "body_md": "If you ship an LLM feature, you probably put a redaction layer in front of the model to strip sensitive data. Good. There's already solid, independent work measuring how well these tools catch **PII** — academic benchmarks like LeakageBench and REDACT, vendor comparisons on datasets like ai4privacy. If PII is your worry, that ground is covered.\n\nHere's what *isn't* measured: **secrets.** The API keys, tokens, connection strings and private keys that end up in prompts not because a user typed them, but because someone pasted a stack trace, an env dump, or a \"why isn't this config working?\" blob. Those are arguably worse to leak than a name — and nobody's independently testing whether your redactor stops them.\n\nSo I built a small, zero-dependency, vendor-neutral tool — **LeakProbe** — that fires evasive secret (and PII) payloads at any redactor and reports what survives. Then I ran it against a real, popular OSS library: `redact-pii`.\n\n`redact-pii` is genuinely good at the job it advertises. Against my corpus it redacted **100% of the classic PII** — emails, credit cards, IBANs, US SSNs — including spaced and lightly obfuscated variants that a naive regex misses. If you picked it to scrub PII, it earns its place.\n\nHere it gets interesting — the coverage is a **patchwork**:\n\n| Secret | Result | \n|---|---|\n| Google API key, Slack token, GitHub fine-grained PAT | Caught | \n| DB / Mongo connection strings (password in the URL) | Caught | \n| **Private keys — RSA, EC, OpenSSH, even well-formed PEM** | **Leaked (100%)** | \n| **Stripe live key, SendGrid key, AWS secret access key** | **Leaked** | \n| **AWS access key id, OpenAI key, classic GitHub PAT** | **Leaked** | \n| **JWT / Bearer token inside a JSON blob** | **Leaked** | \n| Moroccan phone / national ID (CIN) | **Leaked** | \n\nThis is sneakier than \"PII tools are blind to secrets.\" `redact-pii` catches *some* credentials — Google, Slack, GitHub fine-grained tokens, even passwords inside connection strings. That partial coverage is exactly what lulls you into trusting it. Then your **private keys, your Stripe live key, and your AWS secret access key** sail straight through, and you cannot eyeball which half you got. (Overall on my corpus: 20 of 38 payloads leaked, 52.6%.)\n\nIt's not that the library is broken — it's a *PII* tool, and a private key isn't PII. The failure is the **deployment**: assuming one redactor covers everything sensitive going into your prompts.\n\nTwo takeaways:\n\nAnd these aren't hypothetical — the gaps mirror real, documented issues: streamed-output splits ([presidio #2312](https://github.com/data-privacy-stack/presidio/issues/2312)), silently-dropped locale entities ([presidio #2256](https://github.com/data-privacy-stack/presidio/issues/2256)), and secrets hiding in tool-call arguments ([litellm #41265](https://github.com/BerriAI/litellm/issues/41265)). LeakProbe ships a test for each.\n\nLeakProbe is open source, zero dependencies. Write a five-line adapter around whatever you use today and get a per-category leak report.\n\n```\ngit clone https://github.com/roshcompanylabs/leakprobe && cd leakprobe\nnode src/cli.js run --target ./your-adapter.js --threshold 0   # fail CI on any secret that leaks\n```\n\nIf you run it and something surprising leaks, I'd genuinely like to hear what.", "url": "https://wpnews.pro/news/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of", "canonical_source": "https://dev.to/roshcompanylabs/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of-secrets-3ig8", "published_at": "2026-10-23 12:33:03+00:00", "updated_at": "2026-10-23 12:41:51.522326+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "developer-tools", "large-language-models", "ai-tools"], "entities": ["LeakProbe", "redact-pii", "Presidio", "LiteLLM", "Google", "Slack", "GitHub", "Stripe"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of", "markdown": "https://wpnews.pro/news/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of.md", "text": "https://wpnews.pro/news/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of.txt", "jsonld": "https://wpnews.pro/news/your-pii-redactor-is-not-a-secrets-filter-and-your-llm-prompts-are-full-of.jsonld"}}