I needed a deterministic, local way to fail a build the second an LLM leaks a patient identifier, so I built phi-leak-guard
. It's a zero-dependency TypeScript library that integrates directly into Vitest or Jest.
Integration and Usage #
Setting this up as a regression gate in your CI/CD pipeline is straightforward. Since it runs locally, no data ever leaves your process.
import 'phi-leak-guard/vitest';
test('clinical summary never leaks PHI', () => {
expect(summarize(patientNote)).toContainNoPHI();
});
When a leak is detected, the library specifies the identifier type and whether it was a pattern match or a validated hit (e.g., [nhs-number] "943 476 5919" (validated)
).
Why Deterministic Validation Beats Naive Regex #
The biggest pain point in PII/PHI detection is the false positive. A standard 10-digit regex will flag every single order number or timestamp as a sensitive ID. To solve this, the library uses validation logic rather than just pattern matching:
NHS Numbers: Uses Modulus-11 checksums.Vehicle VINs: Implements ISO-3779 check digits.IPv4: Validates octet ranges.SSN/NINO: Applies structural and prefix rules.
If a number doesn't pass the checksum, it isn't flagged. This ensures the build only fails when there is a high probability of an actual leak.
Coverage and Limitations #
This is a risk-reduction tool, not a full compliance certification. It focuses on the 18 HIPAA Safe Harbor identifiers. While it handles direct identifiers well, it cannot detect paraphrased re-identification or every possible name. For those needing deeper analysis, there is a pluggable seam to integrate a Named Entity Recognition (NER) model.
To see exactly what is being monitored, you can run coverageReport()
, which explicitly lists what the library can and cannot detect (e.g., it won't pretend to detect biometrics in plain text).
Deployment #
For anyone building a medical AI workflow, you can add this to your dev dependencies:
npm install --save-dev phi-leak-guard
The library is MIT licensed, supports ESM/CJS, and hits a 0.97 recall on synthetic benchmarks.
https://github.com/selvassn/phi-leak-guard
Next AgentWire: A Wireshark for AI Agents →