# I shipped an npm package that fails your build when an LLM leaks PHI

> Source: <https://dev.to/selvakg/i-shipped-an-npm-package-that-fails-your-build-when-an-llm-leaks-phi-1a65>
> Published: 2026-07-26 11:40:37+00:00

I build LLM features over clinical text. Before shipping any of them, I wanted one boring guarantee: if the model ever puts a patient identifier in its output, the build fails.

I went looking for a tool. Every LLM-eval library I found (autoevals, promptfoo, vitest-evals, evalite) fell into one of two camps: no concept of PHI at all, or it graded output by sending it to a hosted API. The second is a non-starter — when the thing you're evaluating is patient data, mailing it to a third party is the exact risk you're trying to avoid.

So I built phi-leak-guard.

A zero-dependency TypeScript library that detects PHI/PII in text and fails your test suite when it finds any. It runs locally in Vitest/Jest — no network, no model, nothing leaves the process.

``` js
import 'phi-leak-guard/vitest';

 test('clinical summary never leaks PHI', () => {
   expect(summarize(patientNote)).toContainNoPHI();
});
```

When it fails, it tells you exactly what leaked:

Expected output to contain no PHI, but found 2:

The interesting part: precision without a model

The hard problem with deterministic PHI detection is false positives. A naive \d{10} regex flags every order number as an NHS number and gets disabled within a day. The trick is to validate, not just pattern-match:

A random 10-digit number fails the checksum, so it isn't flagged. Matches are tagged validated (checksum-backed) or pattern (regex/context) so you can see how much to trust each hit.

Staying honest about coverage

HIPAA Safe Harbor is a finite list of 18 identifier categories — a completable target. UK GDPR "personal data" is open-ended, so the library covers common direct identifiers, not "everything." A coverageReport() prints exactly what is and isn't checked per standard, and three categories (biometrics, photos, "any other identifier") are reported as not detectable in text rather than pretended-covered.

It's not a compliance certification — it's a regression gate that reduces risk. And deterministic matching can't catch paraphrased re-identification or every name, so there's a pluggable seam to drop in an NER model.

Try it

`npm install --save-dev phi-leak-guard`

It's MIT, ships ESM + CJS + types, and has a synthetic benchmark (precision 1.00, recall 0.97). Repo: [https://github.com/selvassn/phi-leak-guard](https://github.com/selvassn/phi-leak-guard) — I'd genuinely love feedback on which identifiers to add next.