Building localscrub, a local-first PHI de-identification cascade, and benchmarking it honestly against the standard baseline - on one consumer laptop, with zero real patient data.
De-identifying clinical text today forces a bad trade. Cloud de-id APIs are
accurate, but you send the sensitive data out in order to scrub it - the
text crosses your trust boundary before a single character is redacted.
Local rule-based tools keep the data home, but miss exactly the PHI that
matters most: the context-dependent kind. A regex will catch an SSN every
time; it will never catch "the patient's sister works at the bakery on Elm
Street."
localscrub is my attempt to
refuse the trade. It runs a two-stage cascade entirely on your hardware: a
fast rules-and-NER pass for the well-formatted identifiers - phones, emails,
dates, account numbers - and a local LLM, served by Ollama with no network
egress, for the ambiguous remainder. (How much each stage carries is an
empirical question; the benchmark below answers it rather than assuming.)
It's on PyPI as v0.1 (pip install localscrub
), and every number in this
article reproduces from seeds on a single RTX 5080 laptop.
This is the story of building it - and more importantly, of measuring it,
because a privacy tool with unverifiable accuracy claims is just a liability
with a nice README. Along the way: a test set that is a function rather than
a file, a 125-million-parameter model that beat a 14-billion-parameter one,
an eval harness that indicted its own gold standard, and one cursed note
that killed a three-hour benchmark at 99% complete.
You cannot measure a de-identifier without ground truth, and I refused to
use real PHI to get it. The gold-standard clinical de-id corpus (i2b2/n2c2
- sits behind a data use agreement, and scraping a third-party re-upload
would make a privacy project sloppy about data provenance on day one.
So the corpus is generated. localscrub synth
renders synthetic clinical
notes from templates - seven variants across six note types - with
fabricated identifiers planted at recorded character offsets. Every phone
number is from the reserved 555-01XX block, every domain from RFC 2606,
every IP from RFC 5737, every credit card Luhn-valid on a test prefix. The
output is JSONL with exact gold spans, deterministic from a seed:
localscrub synth -n 500 --seed 42 -o eval.jsonl
That determinism buys something subtle: an eval number becomes a property
of the code, not of a dataset file. Corpora are gitignored and
regenerated at will. The test set is a function, not a file.
Template-generated text invites an obvious objection: a detector could
memorize the templates. The answer is --diversify
, which lets a local LLM
paraphrase the connective prose without ever seeing an identifier: every
gold span is masked behind a sentinel token ([[E3]]
), the model rewrites
around the sentinels, values are re-substituted, offsets recomputed. A
rewrite is rejected if any sentinel is dropped or duplicated - or if
re-running stage 1 on the rebuilt text finds identifier-shaped strings
outside the gold spans, i.e. the model invented PHI. The validation loop
cost about ten lines and closes the biggest ground-truth-corruption risk.
That is how you let an LLM touch your test set without trusting it.
The harness (localscrub eval
) reports three numbers per entity type, and
keeping them separate turned out to matter more than any single one:
Redaction recall is the safety metric, and it is deliberately unforgiving:
a detection that leaves half an address in the text does not count. Partial
redaction of an address is still a leak.
The three-metric split earned its keep on the very first run. Stage 1's URL
recognizer scored relaxed 1.00 and strict 0.00 - it was swallowing
sentence-final periods on every single URL. Overlap-only scoring would never
have surfaced it. The same first run put honest zeros on the board: NAME
0.00, GEO 0.00, because stage 1 has no name recognizer by design. Overall
redaction recall: 0.62. That 0.62 turned "stage 2 is on the roadmap" into a
quantified gap - 38% of gold spans unprotected without it.
Stage 2 asks a local model (qwen3:14b via Ollama) to extract
context-dependent PHI. The contract is the highest-leverage decision in the
codebase: the model returns verbatim snippets plus a type - never character offsets. LLMs cannot count characters, but they copy substrings
str.find
instead of corrupting a redaction. Ask the model forThe merge with stage 1 is additive and fail-closed. Stage-1 detections win
overlaps - rules are better calibrated where rules apply. Ambiguous spans
stage 1 flagged are put to the model for adjudication, but only in one
direction: an escalation the model confirms is resolved; one it stays
silent on remains escalated and gets redacted anyway. A 14B model's "no"
never unredacts anything.
First contact, 50 notes: redaction recall 0.62 → 0.89, NAME relaxed recall
0.00 → 0.98. And one open wound: the model found "Cedar Vale" but clipped
"4050 Mossbank Blvd", fully covering only 12% of address spans. Relaxed F1
made GEO look twice as healthy as it was; redaction recall told the truth.
Before reaching for fine-tuning, I tried the boring thing: an off-the-shelf
de-id-specific token classifier (obi/deid_roberta_i2b2
, 125M parameters,
trained on the i2b2 2014 corpus) wrapped as an optional stage-1 recognizer
(pip install 'localscrub[ner]'
). Only its name and location labels are
mapped; dates, phones, and emails stay with the regexes, whose boundaries
are already exact.
On the template corpus it was decisive: redaction recall 0.94 at 184 ms
per note on CPU - matching the 14B model at the categories it was trained
for, roughly forty times faster, no GPU.
Let me concede the framing objection before anyone raises it: a specialist
trained on exactly this task beating a prompted generalist is expected, not
shocking. The finding is the size of the trade - equal recall at
one-fortieth the latency, no GPU - and it matters because the default
recipe today is "throw an LLM at it," and for structured text the default
is measurably wrong. Nor was the 14B handicapped: it ran qwen3:14b at
temperature 0 with schema-constrained decoding and the same
escalation-hint prompt that inference uses
(extraction_prompt
in the repo).
The two models fail differently,
and that mattered later: the LLM copies name boundaries nearly perfectly
but clips addresses; the token classifier covers whole addresses but drags
titles and credentials into name spans. Complementary failure modes,
measurable as such.
Integrating it produced the best debugging afternoon of the project - three
real bugs and a gold-standard flaw, all surfaced by the eval:
name@example.org.\n\nNext Name
became one NAME span).darius.ashcombe@example.com
as a
PATIENT name, because emails literally contain patient names. A person
name never contains @
; drop such spans at the source.Rule of integration, now baked into regression floors: adding a detector
must never make another detector worse.
Templates were too easy, and by this point provably so. The next test
injects synthetic identifiers into ~5,000 authentic public
medical-transcription samples (mtsamples.com - downloaded with a pinned
checksum, never redistributed). Injections are unlabeled narrative
sentences woven between real sentences at deterministic positions:
localscrub mtsamples --fetch -n 100 --seed 42 -o mts.jsonl
One methodological point worth stating plainly: on an injection benchmark,
recall is exact but precision is only a lower bound. The real
transcripts contain their own name-like and date-like strings - "Dr. X"
placeholders, real dates - so a detector flagging them is penalized for
being right. Redaction recall over the injected gold is the number to
trust.
Every configuration, both corpora, all at full size, scored by the
identical harness: 500 template notes carrying 5,021 gold spans, and 100
MTSamples notes carrying 697 injected gold spans - both from seed 42.
Redaction recall - the safety metric - plus precision on the
authentic-prose corpus, where over-flagging shows:
| config | template | MTSamples | MTS precision† | latency/note | hardware |
|---|---|---|---|---|---|
| Presidio (rules-only baseline) | 0.78 | 0.75 | 0.49 | 14–46 ms | CPU |
| stage 1 (rules) | 0.66 | 0.62 | 0.94 | µs | CPU |
| stage 1 + NER | 0.94 | 0.999 | |||
| 0.81 | 184–652 ms | CPU | |||
| stage 1 + LLM | 0.94 | 0.967 | 0.82 | 7–8 s | GPU |
| stage 1 + NER + LLM | 0.94 | 0.999 | |||
| 0.76 | 7–17 s | GPU |
† relaxed precision on the injection benchmark - a lower bound for every
system, per the previous section.
Because three nines invite scrutiny, here are the raw counts behind the
headline number. 0.999 is 696 of 697 injected spans fully redacted
(Wilson 95% CI 0.992–0.9997); one more miss would read 0.997, so treat the
third digit as "one miss in this sample," not a stability claim. And the
one miss deserves naming: in 2034 Harrowgate Rd, Lantern Hill, VT 93695
,
the NER covered the street line ("2034 Harrowgate Rd") and the
state-plus-ZIP ("VT 93695") but dropped the city - "Lantern Hill" leaked
from between two redactions. The
address-clipping failure mode, surviving at the very tail. (The template
0.94, for comparison, is 4,711 of 5,021 - 310 misses; at that sample size
the second digit is doing honest work.)
Two findings, one of them a negative result I think the field under-reports.
On synthetic templates, the LLM buys nothing. Stage 1 + NER,
stage 1 + LLM, and the full cascade all converge at 0.94 redaction recall -
and at 0.97 relaxed F1 - at latencies spanning 184 milliseconds to 17
seconds. The residual 6% is corpus-bound, not detector-bound. If your text
is structured and identifier-dense, a good token classifier is all the
model you need, and it runs on CPU.
On authentic prose, the LLM earns its keep. Bare rules manage 0.62;
adding the LLM lifts that to 0.967; NER+LLM reaches 0.999. The two
detectors compose exactly as the merge was designed to: the LLM still
clips addresses, NER still covers them. But recall is not free - the
precision column tells the other half. The cascade over-flags on narrative
text (0.81 → 0.76 versus NER alone, both lower bounds), and over-redaction
has a real cost in clinical text: every falsely scrubbed token is signal a
downstream reader loses. That trade is why localscrub's model is
review-and-attest rather than fire-and-forget - and note the baseline pays
the same toll, with Presidio at 0.49 precision on this corpus. Recall is
what the LLM buys; know which side of the trade your application needs.
Microsoft's Presidio (rules + spaCy, the standard open-source baseline)
beats bare stage 1 on redaction recall - 0.78 vs 0.66 - because spaCy
gives it person and place names, which stage 1 intentionally defers. It is
also 13–14× faster than our recommended CPU configuration, and its NAME
boundaries are better than our NER extra's (strict F1 0.87 vs 0.73).
Every scoring ambiguity was resolved in the baseline's favor - its unmapped
types still earn redaction credit.
With the NER extra, localscrub wins where a leak hurts most. Presidio never
fully covered a single gold address on either corpus - spaCy tags "Dayton"
but drops "412 Birch Lane" - and it has no MRN or health-plan recognizer,
fully redacting 2–6% of MRNs and ≤15% of plan IDs. localscrub holds those
at 1.00 in every configuration.
A fair question at this point: if a pretrained 125M classifier already hits
0.999, why train anything? Because a token classifier cannot take stage 2's
seat. It tags a fixed label set - ask it about an identifier type it wasn't
trained on and it has no opinion - and it cannot adjudicate the ambiguous
spans stage 1 escalates. Stage 2's contract, verbatim snippets plus types
as JSON, is a conversation, and only an instruction-following model can
hold up its end. The 14B holds it up at seven seconds a note. The question
worth 17 minutes of GPU time is whether a small model can be taught to.
localscrub sft
emits 2,000
training pairs - the exact inference-time stage-2 prompt, escalation hints
included, paired with gold JSON - and a LoRA recipe (r=16, bf16) tunes
Qwen3-1.7B-Base in 17 minutes on the laptop.
The before/after is stark, but not where I expected. The base 1.7B model
produced unparseable output on 35 of 35 notes; the tuned one failed on
0 of 80. The fine-tune's first product is not accuracy - it's
parseability. Accuracy followed: MTSamples redaction recall 0.61 → 0.92.
(Template recall hit a perfect 1.000, which is optimistic by construction -
train and eval share note skeletons; the docs say so.)
The 0.92-vs-0.999 gap against the big-model cascade is a training-data
diversity gap, not a capacity verdict - 2,000 examples from 7 templates
generalize only partway to real prose, and the recipe documents the fix
(mix in MTSamples-injected and diversified notes). The deeper point: the
synthetic corpus is the asset. Data, training, and eval all regenerate from
seeds; the specialist retrains from scratch in under half an hour on
consumer hardware, with no real PHI anywhere in the loop - including the
prompts.
One bad note must not cost you the run. Stage-2 latency is bimodal:
~5 s per note typically, ~50 s when schema-constrained decoding runs away
to the 4,096-token cap - and in each 500-note pass, exactly one template
note (deterministic at temperature 0) stalled the Ollama server for
minutes before returning HTTP 500. The first time, that single note killed
a 2-hour-50-minute benchmark at note ~495 with nothing written, because the
eval had no per-note error handling. The fix - retry the note once, then
score it without stage 2 and report an llm_failures
count - turned the
second occurrence into a 4.8-second retry-and-continue. If you benchmark
local LLMs, build this in before your first long run, not after.
Cap your decoders. Uncapped schema-constrained generation once looped
past a ten-minute timeout. A num_predict
cap plus a salvage parser (parse
the longest well-formed prefix of a truncated extraction list; every item
is independently verified against the source anyway) converts runaway
decoding from a crash into a bounded cost.
Assorted potholes: a gitignore trailing comment silently unignored a
17 MB dataset and it reached the git index once; Blackwell GPUs need cu13x
torch builds and uv needed --reinstall
with an explicit +cu130
pin to
swap them; an untuned base model with no token cap looks exactly like a
frozen process.
obi/deid_roberta_i2b2
was trained on i2b2 2014, not MTSamples; for the LLM's pretraining the honest answer is unknown). The scored identifiers, however, are not MTSamples content: every gold span is synthetic, seeded, and injected - the identifier strings are generated, not drawn from the transcripts, so memorizing MTSamples does not hand a model the answers. What leakage Two measurements are deliberately absent, and they are the next article.
Cloud de-id APIs: the accuracy/latency/cost legs require sending the
benchmark corpora to each provider - synthetic or not, that crossing of the
trust boundary deserves its own explicit decision and write-up, because it
is the exact trade this project exists to interrogate. And i2b2/n2c2 2014: the literature-comparable corpus of naturally occurring PHI, access
Until then: the code is on
GitHub, the package is on
PyPI, and every number above
regenerates from a seed. Check my math.