{"slug": "i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi", "title": "I shipped an npm package that fails your build when an LLM leaks PHI", "summary": "A developer built phi-leak-guard, a zero-dependency TypeScript library that detects PHI/PII in text and fails test suites locally in Vitest/Jest without sending data to any external API. The library uses checksum-validated pattern matching to avoid false positives and covers HIPAA Safe Harbor identifiers with a pluggable seam for NER models.", "body_md": "I build LLM features over clinical text. Before shipping any of them, I wanted one boring guarantee: if the model ever puts a patient identifier in its output, the build fails.\n\nI went looking for a tool. Every LLM-eval library I found (autoevals, promptfoo, vitest-evals, evalite) fell into one of two camps: no concept of PHI at all, or it graded output by sending it to a hosted API. The second is a non-starter — when the thing you're evaluating is patient data, mailing it to a third party is the exact risk you're trying to avoid.\n\nSo I built phi-leak-guard.\n\nA zero-dependency TypeScript library that detects PHI/PII in text and fails your test suite when it finds any. It runs locally in Vitest/Jest — no network, no model, nothing leaves the process.\n\n``` js\nimport 'phi-leak-guard/vitest';\n\n test('clinical summary never leaks PHI', () => {\n   expect(summarize(patientNote)).toContainNoPHI();\n});\n```\n\nWhen it fails, it tells you exactly what leaked:\n\nExpected output to contain no PHI, but found 2:\n\nThe interesting part: precision without a model\n\nThe hard problem with deterministic PHI detection is false positives. A naive \\d{10} regex flags every order number as an NHS number and gets disabled within a day. The trick is to validate, not just pattern-match:\n\nA random 10-digit number fails the checksum, so it isn't flagged. Matches are tagged validated (checksum-backed) or pattern (regex/context) so you can see how much to trust each hit.\n\nStaying honest about coverage\n\nHIPAA Safe Harbor is a finite list of 18 identifier categories — a completable target. UK GDPR \"personal data\" is open-ended, so the library covers common direct identifiers, not \"everything.\" A coverageReport() prints exactly what is and isn't checked per standard, and three categories (biometrics, photos, \"any other identifier\") are reported as not detectable in text rather than pretended-covered.\n\nIt's not a compliance certification — it's a regression gate that reduces risk. And deterministic matching can't catch paraphrased re-identification or every name, so there's a pluggable seam to drop in an NER model.\n\nTry it\n\n`npm install --save-dev phi-leak-guard`\n\nIt's MIT, ships ESM + CJS + types, and has a synthetic benchmark (precision 1.00, recall 0.97). Repo: [https://github.com/selvassn/phi-leak-guard](https://github.com/selvassn/phi-leak-guard) — I'd genuinely love feedback on which identifiers to add next.", "url": "https://wpnews.pro/news/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi", "canonical_source": "https://dev.to/selvakg/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi-1a65", "published_at": "2026-07-26 11:40:37+00:00", "updated_at": "2026-07-26 11:59:10.461463+00:00", "lang": "en", "topics": ["developer-tools", "ai-safety", "artificial-intelligence"], "entities": ["phi-leak-guard", "Vitest", "Jest", "HIPAA", "NHS"], "alternates": {"html": "https://wpnews.pro/news/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi", "markdown": "https://wpnews.pro/news/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi.md", "text": "https://wpnews.pro/news/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi.txt", "jsonld": "https://wpnews.pro/news/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi.jsonld"}}