What building a medical transcription pipeline taught me about the difference between a model being sure, and a model being right.
The 10x
A patient was discharged to a rehab facility on an insulin dosage of 80 units. The prescriber had said 8.
Somewhere between the clinician's voice and the chart, a speech-recognition system heard "eighty" instead of "eight," transcribed it with total confidence, and nobody caught it before it reached the patient. This isn't a hypothetical — it's a documented patient safety incident reported through ISMP's medication error tracking. A single digit, silently wrong, and a system that never once flagged it as uncertain.
It's worth sitting with that for a second before we talk about architecture, because it reframes the entire problem. This isn't a story about a model making a mistake. Models make mistakes constantly, and that's fine — errors are recoverable if you can see them. This is a story about a model making a mistake with total confidence, which is what makes it unrecoverable. The failure wasn't the wrong number. The failure was that nothing in the system knew it should have been unsure.
And it's not an isolated case. A study analyzing emergency department notes generated with speech recognition found an average of 1.3 errors per note, with 15% of those errors judged clinically significant. Another comparison found physician notes produced with speech recognition carried four times the error rate of notes produced without it. In a less severe but still telling example, one transcription recorded a patient's allergy as "XOPENEX" — a bronchodilator — when the clinician had actually said "sulfa." Different drug class. Different allergen. Same silent confidence.
Before automated speech recognition, a human medical transcriptionist filled a role that's easy to overlook until it's gone: they caught the things that were obviously, contextually wrong — a report mentioning "ovaries" for a male patient, say — and flagged it before it reached the chart. When software replaced that role, it didn't replace that instinct. It just removed the checkpoint. Some clinical records now carry a standing disclaimer warning that "wrong-word or sound-a-like substitutions may have occurred" and asking readers to catch them manually — which is, functionally, the industry admitting it never solved this problem. It just pushed it back onto humans and hoped they'd stay vigilant forever.
MedScribe, the clinical documentation pipeline I built, sits in exactly this gap — converting patient-visit audio into corrected transcripts and structured notes using offline ASR (FasterWhisper) and LLM-based medical term extraction.
It didn't start out this careful. The first version, back in early 2025, was honestly just Whisper wrapped in a UI — transcribe the audio, show the text, done. It worked, in the sense that a demo works. It took actually digging into the failure literature above, and watching my own pipeline confidently mangle "atorva statin" into the wrong thing, to understand that the interesting problem wasn't transcription accuracy at all. It was: how does the model know when it isn't sure enough to touch a medical term at all?
Why "just set a confidence threshold" doesn't work
The obvious first instinct — mine included — is to just check the model's confidence score. High confidence, auto-correct. Low confidence, flag for review. Done.
Except confidence scores from neural networks don't mean what they look like they mean. A model is calibrated if, among everything it labels "70% confident," roughly 70% actually turns out correct. Real models routinely violate this. They're overconfident — assigning high certainty to predictions that are simply wrong.
Here's the number that should worry anyone building on ASR: a 2025 study analyzing Whisper — the same model family behind MedScribe's transcription layer — found that under noisy conditions, 10–20% of tokens were incorrectly predicted with confidence above 0.7. And critically, the same study found overconfident errors increase specifically as signal-to-noise ratio drops. That's not a random failure mode. That's the threshold breaking exactly where you need it most — in a real exam room, with background chatter and equipment hum, not a quiet recording studio.
This isn't an ASR-specific quirk either. It's a well-documented property of deep neural networks broadly — across vision, structured prediction, and language tasks, models are known to assign high confidence to samples they get wrong. Even large language models show it: one evaluation of verbalized uncertainty found models clustering almost all their stated confidence in the 90–100% range regardless of whether they were actually right, with one model's expected calibration error measured at 74.8% — meaning its stated confidence was almost completely disconnected from its real accuracy.
In a clinical pipeline, a single threshold fails in two directions at once, and both are bad. Set it too loose, and you get exactly the insulin scenario — a confidently wrong correction slipping straight through. Set it too strict, and you flag so many correct terms as "uncertain" that clinicians drown in false alarms and start ignoring the flags altogether — a well-known failure mode in clinical software called alert fatigue. You can't fix a two-sided problem by moving a single number back and forth.
I don't have to imagine what this looks like — my own audit logs are full of it. Running MedScribe's synthetic benchmark corpus through fault-tree analysis turned up exactly the sound-alike corruptions the research predicts: citigliptin for Sitagliptin, listen April 20 MG for Lisinopril 20mg, Fluticusone for Fluticasone, NSA IDS for NSAIDs, torvostat for Atorvastatin. Every one of these is a case where a naive similarity check alone has to decide, in isolation, whether it's looking at a real drug name or noise — and similarity scores alone don't carry enough information to tell the difference reliably.
The clearest lesson came from a mistake in my own evaluation setup, not the model. I renamed a canonical dictionary entry from Metformin Hcl to Metformin, and my hand-labeled eval case still expected the old name — so my "wrong auto-correct rate" metric spiked from 0% to 20% overnight. The system had actually corrected metform in to Metformin perfectly. The label was stale, not the model. It's a small story, but it's the honest version of "20% to 0%" — the naive read of a single confusion-matrix number can be just as misleading as a naive confidence threshold, if you don't dig into why it moved.
Two signals, not one: the AND-gate This kind of problem has a name in the research literature I didn't know when I started: selective prediction, sometimes called abstention — systems that are allowed to withhold a decision rather than commit to one that might be wrong. It traces back to Chow's 1970 "reject option" in pattern recognition, and it shows up explicitly in medical AI research today, where the most uncertain predictions in a diagnostic pipeline get routed to a clinician for review instead of being acted on automatically. MedScribe's correction engine follows the same shape, just applied to transcription instead of diagnosis.
Recent work on abstention is pretty direct about the weakness of single-signal designs: most methods derive their decision from one confidence score or a hand-picked threshold, rather than a more rigorous check for whether the prediction is actually reliable. That's precisely the gap I needed to close.
So MedScribe's correction engine checks three independent signals, not two, and combines them in a specific order rather than a flat AND of everything:
Retrieval similarity — how close is the candidate term to a known entry in the medical dictionary (via pgvector, against canonical terms and aliases)?
Extraction confidence — how sure was the LLM, in isolation, that it identified the right term from the raw transcript?
Margin — of the candidate's matches, how much daylight is there between the top hit and the runner-up?
Similarity and confidence form the first gate: a correction is only eligible to auto-apply if similarity clears 0.90 and extraction confidence clears 0.70. That already rules out two different failure modes independently — a retrieval near-miss with a confident extractor, or a strong dictionary match paired with an extractor that wasn't sure what it was looking at.
But eligibility isn't enough on its own, because a candidate can clear both bars while still being ambiguous between two plausible dictionary entries — the "sulfa" vs. "XOPENEX" problem from earlier. That's what margin catches: if the top match and the runner-up are within 0.05 similarity of each other, the correction is held for review instead of applied, even though it passed both earlier checks. Research on margin-based selective prediction backs this pattern directly: even without a learned classifier on top, the raw gap between a model's top predictions ranks ambiguity almost as well as far more complex uncertainty methods.
There's a fourth check that runs before any of this: identity protection. If the raw term already exactly matches a canonical name or alias, it needs no correction at all — the engine short-circuits straight to a logged identity_protected decision rather than running it through similarity math it doesn't need. It's a small thing, but it means the system isn't manufacturing risk on terms that were already correct.
Every candidate that isn't identity-protected lands in one of five logged outcomes: corrected (both gates and margin passed), review_required (both gates passed, margin didn't — genuinely ambiguous), flagged for high similarity but low confidence, flagged again for moderate similarity below the auto-correct bar, or ignored when nothing matches well enough to act on at all. Two different reasons can produce the same "flagged" label, and the audit log keeps them distinguishable — which matters later, because "the extractor wasn't sure" and "the dictionary just doesn't have a good match" call for completely different fixes.
The reasoning behind requiring multiple independent conditions, rather than one number doing all the work, is the same logic as two-factor authentication: a password alone can be phished or guessed, so you require a password and a one-time code, and compromising just one factor isn't enough to get through. A wrong retrieval match, a low-confidence extraction, and a genuine near-tie are three different ways this can go wrong — so three different checks, each aimed at exactly one of them, catch more than any single number stretched to cover all three.
It's also worth explaining why "just raise the threshold" doesn't substitute for this. Pushing a single confidence bar higher to be "safer" doesn't fix the underlying problem — it just shifts it into more false alarms and more alert fatigue, the same failure mode from the naive approach, just relocated. Independent, purpose-built conditions let you tune each failure mode separately instead of asking one number to do three incompatible jobs at once.
What actually changed
Running the full pipeline against a 25-case synthetic benchmark — spanning ASR corruption, sound-alike drug pairs, and deliberate dictionary coverage gaps — 68 of 83 logged extraction items made it end-to-end correctly, roughly 82%. More importantly for this piece: zero of them were a silently wrong correction. Every miss showed up as correction_failed — visibly wrong or visibly unresolved — never as a confident, undetected error slipped into the note. That's the actual goal. Not perfect accuracy; an honest failure mode.
I'd rather be direct about what that number does and doesn't prove, because overclaiming it would undercut the whole point of this post. My own technical-debt notes say it plainly: the thresholds behind all of this — 0.90 similarity, 0.70 confidence, 0.05 margin — are engineering-judgment defaults, never validated against real outcome data. They came from reading the failure literature and testing against a 25-case synthetic corpus, not from a large sample of real clinical corrections reviewed by an actual clinician. That's a meaningfully different claim than "this is calibrated," and I'd rather say so than let a resume bullet imply more than the eval actually supports.
What I'm confident the design achieves is narrower and, I think, more defensible: it eliminates one specific failure mode — a model that's both confident and wrong, and looks like neither — by refusing to let one number stand in for three different questions. It's trivial to eliminate false corrections by simply flagging everything and refusing to act — but that just relocates the problem into alert fatigue. The 82% end-to-end rate is evidence the system isn't achieving safety by becoming useless; it's still doing real work, just without pretending to certainty it doesn't have.
Beyond one project
None of this is unique to medical transcription, and I think that's the more interesting takeaway. Any system where an AI model takes an action on someone's behalf — an agent calling a tool, an assistant sending an email, a pipeline correcting a financial transaction — has the exact same underlying question sitting under it: not "is the model usually right," but "does the model know, on this specific decision, whether it should act or defer?"
The uncomfortable finding across the research is that models are bad at reporting this about themselves by default. High stated confidence and high actual reliability are two different things, and conflating them is exactly what makes confidently-wrong outputs dangerous instead of just occasionally imperfect. The fix isn't a smarter model — it's an honest second opinion built into the system: a margin check, a disagreement signal, a second independent measure that has to agree before the system commits to an action nobody's watching in real time.
If I were extending this further, the next thing I'd test is whether the same multi-signal pattern holds up for agentic tool-calling decisions — not "is this the right medical term," but "is this the right tool to call, with these parameters, right now." My hypothesis is that it generalizes cleanly, because the underlying failure — a model being sure without being right — doesn't care what kind of decision it's attached to. I'll also just say plainly what I wouldn't ship to a real pilot yet, because a design like this is only as trustworthy as the honesty around its limits: the thresholds need validation against real reviewer outcomes, not just a synthetic benchmark; the eval set itself is still small enough (7–25 cases) that any single mislabeled case swings the headline metric noticeably, as the Metformin incident showed; and there's no PHI handling policy yet for the raw transcripts sitting in the audit log — which, for anything touching real patient data, is a blocker before it's a nice-to-have. None of these undermine the core idea. They're exactly the kind of thing selective prediction research doesn't advertise on the tin: the abstention mechanism can be sound and the system can still not be ready, because trust has to be earned at every layer, not just the one you wrote a blog post about.
MedScribe is a full-stack clinical documentation pipeline using offline ASR (FasterWhisper), LLM-based medical term extraction, and pgvector similarity retrieval, with every correction decision logged and reconstructible from the audit trail.