I build LLM features over clinical text. Before shipping any of them, I wanted one boring guarantee: if the model ever puts a patient identifier in its output, the build fails.
I went looking for a tool. Every LLM-eval library I found (autoevals, promptfoo, vitest-evals, evalite) fell into one of two camps: no concept of PHI at all, or it graded output by sending it to a hosted API. The second is a non-starter — when the thing you're evaluating is patient data, mailing it to a third party is the exact risk you're trying to avoid.
So I built phi-leak-guard.
A zero-dependency TypeScript library that detects PHI/PII in text and fails your test suite when it finds any. It runs locally in Vitest/Jest — no network, no model, nothing leaves the process.
import 'phi-leak-guard/vitest';
test('clinical summary never leaks PHI', () => {
expect(summarize(patientNote)).toContainNoPHI();
});
When it fails, it tells you exactly what leaked:
Expected output to contain no PHI, but found 2:
The interesting part: precision without a model
The hard problem with deterministic PHI detection is false positives. A naive \d{10} regex flags every order number as an NHS number and gets disabled within a day. The trick is to validate, not just pattern-match:
A random 10-digit number fails the checksum, so it isn't flagged. Matches are tagged validated (checksum-backed) or pattern (regex/context) so you can see how much to trust each hit.
Staying honest about coverage
HIPAA Safe Harbor is a finite list of 18 identifier categories — a completable target. UK GDPR "personal data" is open-ended, so the library covers common direct identifiers, not "everything." A coverageReport() prints exactly what is and isn't checked per standard, and three categories (biometrics, photos, "any other identifier") are reported as not detectable in text rather than pretended-covered.
It's not a compliance certification — it's a regression gate that reduces risk. And deterministic matching can't catch paraphrased re-identification or every name, so there's a pluggable seam to drop in an NER model.
Try it
npm install --save-dev phi-leak-guard
It's MIT, ships ESM + CJS + types, and has a synthetic benchmark (precision 1.00, recall 0.97). Repo: https://github.com/selvassn/phi-leak-guard — I'd genuinely love feedback on which identifiers to add next.