{"slug": "a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40-faster-on-cpu", "title": "A 125M model beat a 14B LLM at de-identifying medical text 40 faster, on CPU", "summary": "A developer built localscrub, a local-first PHI de-identification cascade that runs entirely on consumer hardware, and benchmarked it against standard baselines. The system uses a two-stage pipeline combining rules and NER with a local LLM served by Ollama, and a 125M-parameter model outperformed a 14B model while running 40 times faster on CPU. The project includes a synthetic test set generator and an eval harness that revealed critical issues in existing metrics.", "body_md": "*Building localscrub, a local-first PHI de-identification cascade, and\nbenchmarking it honestly against the standard baseline - on one consumer\nlaptop, with zero real patient data.*\n\nDe-identifying clinical text today forces a bad trade. Cloud de-id APIs are\n\naccurate, but you send the sensitive data out in order to scrub it - the\n\ntext crosses your trust boundary before a single character is redacted.\n\nLocal rule-based tools keep the data home, but miss exactly the PHI that\n\nmatters most: the context-dependent kind. A regex will catch an SSN every\n\ntime; it will never catch \"the patient's sister works at the bakery on Elm\n\nStreet.\"\n\n[localscrub](https://github.com/valbarov/localscrub) is my attempt to\n\nrefuse the trade. It runs a two-stage cascade entirely on your hardware: a\n\nfast rules-and-NER pass for the well-formatted identifiers - phones, emails,\n\ndates, account numbers - and a local LLM, served by Ollama with no network\n\negress, for the ambiguous remainder. (How much each stage carries is an\n\nempirical question; the benchmark below answers it rather than assuming.)\n\nIt's on PyPI as v0.1 (`pip install localscrub`\n\n), and every number in this\n\narticle reproduces from seeds on a single RTX 5080 laptop.\n\nThis is the story of building it - and more importantly, of *measuring* it,\n\nbecause a privacy tool with unverifiable accuracy claims is just a liability\n\nwith a nice README. Along the way: a test set that is a function rather than\n\na file, a 125-million-parameter model that beat a 14-billion-parameter one,\n\nan eval harness that indicted its own gold standard, and one cursed note\n\nthat killed a three-hour benchmark at 99% complete.\n\nYou cannot measure a de-identifier without ground truth, and I refused to\n\nuse real PHI to get it. The gold-standard clinical de-id corpus (i2b2/n2c2\n\n2014) sits behind a data use agreement, and scraping a third-party re-upload\n\nwould make a privacy project sloppy about data provenance on day one.\n\nSo the corpus is generated. `localscrub synth`\n\nrenders synthetic clinical\n\nnotes from templates - seven variants across six note types - with\n\nfabricated identifiers planted at recorded character offsets. Every phone\n\nnumber is from the reserved 555-01XX block, every domain from RFC 2606,\n\nevery IP from RFC 5737, every credit card Luhn-valid on a test prefix. The\n\noutput is JSONL with exact gold spans, deterministic from a seed:\n\n```\nlocalscrub synth -n 500 --seed 42 -o eval.jsonl\n```\n\nThat determinism buys something subtle: an eval number becomes a property\n\nof the *code*, not of a dataset file. Corpora are gitignored and\n\nregenerated at will. The test set is a function, not a file.\n\nTemplate-generated text invites an obvious objection: a detector could\n\nmemorize the templates. The answer is `--diversify`\n\n, which lets a local LLM\n\nparaphrase the connective prose *without ever seeing an identifier*: every\n\ngold span is masked behind a sentinel token (`[[E3]]`\n\n), the model rewrites\n\naround the sentinels, values are re-substituted, offsets recomputed. A\n\nrewrite is rejected if any sentinel is dropped or duplicated - or if\n\nre-running stage 1 on the rebuilt text finds identifier-shaped strings\n\noutside the gold spans, i.e. the model *invented* PHI. The validation loop\n\ncost about ten lines and closes the biggest ground-truth-corruption risk.\n\nThat is how you let an LLM touch your test set without trusting it.\n\nThe harness (`localscrub eval`\n\n) reports three numbers per entity type, and\n\nkeeping them separate turned out to matter more than any single one:\n\nRedaction recall is the safety metric, and it is deliberately unforgiving:\n\na detection that leaves half an address in the text does not count. Partial\n\nredaction of an address is still a leak.\n\nThe three-metric split earned its keep on the very first run. Stage 1's URL\n\nrecognizer scored relaxed 1.00 and strict **0.00** - it was swallowing\n\nsentence-final periods on every single URL. Overlap-only scoring would never\n\nhave surfaced it. The same first run put honest zeros on the board: NAME\n\n0.00, GEO 0.00, because stage 1 has no name recognizer by design. Overall\n\nredaction recall: 0.62. That 0.62 turned \"stage 2 is on the roadmap\" into a\n\nquantified gap - 38% of gold spans unprotected without it.\n\nStage 2 asks a local model (qwen3:14b via Ollama) to extract\n\ncontext-dependent PHI. The contract is the highest-leverage decision in the\n\ncodebase: the model returns **verbatim snippets plus a type - never\ncharacter offsets**. LLMs cannot count characters, but they copy substrings\n\n`str.find`\n\ninstead of corrupting a redaction. Ask the model forThe merge with stage 1 is additive and fail-closed. Stage-1 detections win\n\noverlaps - rules are better calibrated where rules apply. Ambiguous spans\n\nstage 1 flagged are put to the model for adjudication, but only in one\n\ndirection: an escalation the model confirms is resolved; one it stays\n\nsilent on remains escalated and gets redacted anyway. A 14B model's \"no\"\n\nnever unredacts anything.\n\nFirst contact, 50 notes: redaction recall 0.62 → 0.89, NAME relaxed recall\n\n0.00 → 0.98. And one open wound: the model found \"Cedar Vale\" but clipped\n\n\"4050 Mossbank Blvd\", fully covering only 12% of address spans. Relaxed F1\n\nmade GEO look twice as healthy as it was; redaction recall told the truth.\n\nBefore reaching for fine-tuning, I tried the boring thing: an off-the-shelf\n\nde-id-specific token classifier (`obi/deid_roberta_i2b2`\n\n, 125M parameters,\n\ntrained on the i2b2 2014 corpus) wrapped as an optional stage-1 recognizer\n\n(`pip install 'localscrub[ner]'`\n\n). Only its name and location labels are\n\nmapped; dates, phones, and emails stay with the regexes, whose boundaries\n\nare already exact.\n\nOn the template corpus it was decisive: redaction recall 0.94 at 184 ms\n\nper note on CPU - matching the 14B model at the categories it was trained\n\nfor, roughly forty times faster, no GPU.\n\nLet me concede the framing objection before anyone raises it: a specialist\n\ntrained on exactly this task beating a prompted generalist is expected, not\n\nshocking. The finding is the *size of the trade* - equal recall at\n\none-fortieth the latency, no GPU - and it matters because the default\n\nrecipe today is \"throw an LLM at it,\" and for structured text the default\n\nis measurably wrong. Nor was the 14B handicapped: it ran qwen3:14b at\n\ntemperature 0 with schema-constrained decoding and the same\n\nescalation-hint prompt that inference uses\n\n(`extraction_prompt`\n\nin the repo).\n\nThe two models fail *differently*,\n\nand that mattered later: the LLM copies name boundaries nearly perfectly\n\nbut clips addresses; the token classifier covers whole addresses but drags\n\ntitles and credentials into name spans. Complementary failure modes,\n\nmeasurable as such.\n\nIntegrating it produced the best debugging afternoon of the project - three\n\nreal bugs and a gold-standard flaw, all surfaced by the eval:\n\n`name@example.org.\\n\\nNext Name`\n\nbecame one NAME span).`darius.ashcombe@example.com`\n\nas a\nPATIENT name, because emails literally contain patient names. A person\nname never contains `@`\n\n; drop such spans at the source.Rule of integration, now baked into regression floors: adding a detector\n\nmust never make another detector worse.\n\nTemplates were too easy, and by this point provably so. The next test\n\ninjects synthetic identifiers into ~5,000 authentic public\n\nmedical-transcription samples (mtsamples.com - downloaded with a pinned\n\nchecksum, never redistributed). Injections are unlabeled narrative\n\nsentences woven between real sentences at deterministic positions:\n\n```\nlocalscrub mtsamples --fetch -n 100 --seed 42 -o mts.jsonl\n```\n\nOne methodological point worth stating plainly: on an injection benchmark,\n\n**recall is exact but precision is only a lower bound.** The real\n\ntranscripts contain their own name-like and date-like strings - \"Dr. X\"\n\nplaceholders, real dates - so a detector flagging them is penalized for\n\nbeing right. Redaction recall over the injected gold is the number to\n\ntrust.\n\nEvery configuration, both corpora, all at full size, scored by the\n\nidentical harness: 500 template notes carrying 5,021 gold spans, and 100\n\nMTSamples notes carrying 697 injected gold spans - both from seed 42.\n\nRedaction recall - the safety metric - plus precision on the\n\nauthentic-prose corpus, where over-flagging shows:\n\n| config | template | MTSamples | MTS precision† | latency/note | hardware |\n|---|---|---|---|---|---|\n| Presidio (rules-only baseline) | 0.78 | 0.75 | 0.49 | 14–46 ms | CPU |\n| stage 1 (rules) | 0.66 | 0.62 | 0.94 | µs | CPU |\n| stage 1 + NER | 0.94 | 0.999 |\n0.81 | 184–652 ms | CPU |\n| stage 1 + LLM | 0.94 | 0.967 | 0.82 | 7–8 s | GPU |\n| stage 1 + NER + LLM | 0.94 | 0.999 |\n0.76 | 7–17 s | GPU |\n\n† relaxed precision on the injection benchmark - a lower bound for every\n\nsystem, per the previous section.\n\nBecause three nines invite scrutiny, here are the raw counts behind the\n\nheadline number. 0.999 is **696 of 697** injected spans fully redacted\n\n(Wilson 95% CI 0.992–0.9997); one more miss would read 0.997, so treat the\n\nthird digit as \"one miss in this sample,\" not a stability claim. And the\n\none miss deserves naming: in `2034 Harrowgate Rd, Lantern Hill, VT 93695`\n\n,\n\nthe NER covered the street line (\"2034 Harrowgate Rd\") and the\n\nstate-plus-ZIP (\"VT 93695\") but dropped the city - \"Lantern Hill\" leaked\n\nfrom between two redactions. The\n\naddress-clipping failure mode, surviving at the very tail. (The template\n\n0.94, for comparison, is 4,711 of 5,021 - 310 misses; at that sample size\n\nthe second digit is doing honest work.)\n\nTwo findings, one of them a negative result I think the field under-reports.\n\n**On synthetic templates, the LLM buys nothing.** Stage 1 + NER,\n\nstage 1 + LLM, and the full cascade all converge at 0.94 redaction recall -\n\nand at 0.97 relaxed F1 - at latencies spanning 184 milliseconds to 17\n\nseconds. The residual 6% is corpus-bound, not detector-bound. If your text\n\nis structured and identifier-dense, a good token classifier is all the\n\nmodel you need, and it runs on CPU.\n\n**On authentic prose, the LLM earns its keep.** Bare rules manage 0.62;\n\nadding the LLM lifts that to 0.967; NER+LLM reaches 0.999. The two\n\ndetectors compose exactly as the merge was designed to: the LLM still\n\nclips addresses, NER still covers them. But recall is not free - the\n\nprecision column tells the other half. The cascade over-flags on narrative\n\ntext (0.81 → 0.76 versus NER alone, both lower bounds), and over-redaction\n\nhas a real cost in clinical text: every falsely scrubbed token is signal a\n\ndownstream reader loses. That trade is why localscrub's model is\n\nreview-and-attest rather than fire-and-forget - and note the baseline pays\n\nthe same toll, with Presidio at 0.49 precision on this corpus. Recall is\n\nwhat the LLM buys; know which side of the trade your application needs.\n\nMicrosoft's Presidio (rules + spaCy, the standard open-source baseline)\n\n**beats bare stage 1** on redaction recall - 0.78 vs 0.66 - because spaCy\n\ngives it person and place names, which stage 1 intentionally defers. It is\n\nalso 13–14× faster than our recommended CPU configuration, and its NAME\n\nboundaries are *better* than our NER extra's (strict F1 0.87 vs 0.73).\n\nEvery scoring ambiguity was resolved in the baseline's favor - its unmapped\n\ntypes still earn redaction credit.\n\nWith the NER extra, localscrub wins where a leak hurts most. Presidio never\n\nfully covered a single gold address on either corpus - spaCy tags \"Dayton\"\n\nbut drops \"412 Birch Lane\" - and it has no MRN or health-plan recognizer,\n\nfully redacting 2–6% of MRNs and ≤15% of plan IDs. localscrub holds those\n\nat 1.00 in every configuration.\n\nA fair question at this point: if a pretrained 125M classifier already hits\n\n0.999, why train anything? Because a token classifier cannot take stage 2's\n\nseat. It tags a fixed label set - ask it about an identifier type it wasn't\n\ntrained on and it has no opinion - and it cannot adjudicate the ambiguous\n\nspans stage 1 escalates. Stage 2's contract, verbatim snippets plus types\n\nas JSON, is a conversation, and only an instruction-following model can\n\nhold up its end. The 14B holds it up at seven seconds a note. The question\n\nworth 17 minutes of GPU time is whether a small model can be *taught* to.\n\n`localscrub sft`\n\nemits 2,000\n\ntraining pairs - the exact inference-time stage-2 prompt, escalation hints\n\nincluded, paired with gold JSON - and a LoRA recipe (r=16, bf16) tunes\n\nQwen3-1.7B-Base in 17 minutes on the laptop.\n\nThe before/after is stark, but not where I expected. The base 1.7B model\n\nproduced unparseable output on **35 of 35** notes; the tuned one failed on\n\n0 of 80. The fine-tune's first product is not accuracy - it's\n\n*parseability*. Accuracy followed: MTSamples redaction recall 0.61 → 0.92.\n\n(Template recall hit a perfect 1.000, which is optimistic by construction -\n\ntrain and eval share note skeletons; the docs say so.)\n\nThe 0.92-vs-0.999 gap against the big-model cascade is a training-data\n\ndiversity gap, not a capacity verdict - 2,000 examples from 7 templates\n\ngeneralize only partway to real prose, and the recipe documents the fix\n\n(mix in MTSamples-injected and diversified notes). The deeper point: the\n\nsynthetic corpus is the asset. Data, training, and eval all regenerate from\n\nseeds; the specialist retrains from scratch in under half an hour on\n\nconsumer hardware, with no real PHI anywhere in the loop - including the\n\nprompts.\n\n**One bad note must not cost you the run.** Stage-2 latency is bimodal:\n\n~5 s per note typically, ~50 s when schema-constrained decoding runs away\n\nto the 4,096-token cap - and in each 500-note pass, exactly one template\n\nnote (deterministic at temperature 0) stalled the Ollama server for\n\nminutes before returning HTTP 500. The first time, that single note killed\n\na 2-hour-50-minute benchmark at note ~495 with nothing written, because the\n\neval had no per-note error handling. The fix - retry the note once, then\n\nscore it without stage 2 and report an `llm_failures`\n\ncount - turned the\n\nsecond occurrence into a 4.8-second retry-and-continue. If you benchmark\n\nlocal LLMs, build this in before your first long run, not after.\n\n**Cap your decoders.** Uncapped schema-constrained generation once looped\n\npast a ten-minute timeout. A `num_predict`\n\ncap plus a salvage parser (parse\n\nthe longest well-formed prefix of a truncated extraction list; every item\n\nis independently verified against the source anyway) converts runaway\n\ndecoding from a crash into a bounded cost.\n\n**Assorted potholes:** a gitignore *trailing comment* silently unignored a\n\n17 MB dataset and it reached the git index once; Blackwell GPUs need cu13x\n\ntorch builds and uv needed `--reinstall`\n\nwith an explicit `+cu130`\n\npin to\n\nswap them; an untuned base model with no token cap looks exactly like a\n\nfrozen process.\n\n`obi/deid_roberta_i2b2`\n\nwas trained on i2b2 2014, not MTSamples; for\nthe LLM's pretraining the honest answer is unknown). The scored\nidentifiers, however, are not MTSamples content: every gold span is\nsynthetic, seeded, and injected - the identifier strings are generated,\nnot drawn from the transcripts, so memorizing MTSamples does not hand a\nmodel the answers. What leakage Two measurements are deliberately absent, and they are the next article.\n\n**Cloud de-id APIs**: the accuracy/latency/cost legs require sending the\n\nbenchmark corpora to each provider - synthetic or not, that crossing of the\n\ntrust boundary deserves its own explicit decision and write-up, because it\n\nis the exact trade this project exists to interrogate. And **i2b2/n2c2\n2014**: the literature-comparable corpus of naturally occurring PHI, access\n\nUntil then: the code is on\n\n[GitHub](https://github.com/valbarov/localscrub), the package is on\n\n[PyPI](https://pypi.org/project/localscrub/), and every number above\n\nregenerates from a seed. Check my math.", "url": "https://wpnews.pro/news/a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40-faster-on-cpu", "canonical_source": "https://dev.to/vadim_albarov/a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40x-faster-on-cpu-201a", "published_at": "2026-08-02 04:13:27+00:00", "updated_at": "2026-08-02 05:09:20.287696+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools", "developer-tools"], "entities": ["localscrub", "Ollama", "RTX 5080", "i2b2/n2c2 2014", "PyPI"], "alternates": {"html": "https://wpnews.pro/news/a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40-faster-on-cpu", "markdown": "https://wpnews.pro/news/a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40-faster-on-cpu.md", "text": "https://wpnews.pro/news/a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40-faster-on-cpu.txt", "jsonld": "https://wpnews.pro/news/a-125m-model-beat-a-14b-llm-at-de-identifying-medical-text-40-faster-on-cpu.jsonld"}}