A common assumption in medical AI is that a managed cloud speech service sets the accuracy ceiling, and that self-hosting trades accuracy for privacy. For non-English medical transcription, our results prove the assumption to be wrong.
On a German consultation set, AWS Transcribe reaches 0.82 medical-term recall. A single self-hosted speech model reaches 0.75, as expected for a general open model. A layered self-hosted pipeline reaches 0.91, above the cloud baseline, while keeping all audio on hardware we control.
The result rests on a large evaluation rather than a handful of clips: 5,504 synthetic transcriptions across four models, a set of dense multi-term utterances, and five real consultations as a real-world anchor. Reaching 0.91 took a layered architecture with four deliberate design choices, the first of which is counterintuitive: the German-tuned Whisper model was the weakest starting point and the hardest to improve.
We build an ambient medical scribe for German practices. During a consultation the system listens, transcribes, and produces a structured clinical note and billing codes. Two constraints defined the design space.
First, the system runs on-premise. Audio cannot leave the practice, which rules out any cloud transcription service and includes small single-physician installations.
Second, transcription errors in this domain are clinically significant rather than cosmetic. A drug name misheard as a similar-sounding non-word is not extracted by downstream processing, is not coded for billing, and in the worst case contributes to a medication-reconciliation error.
Accuracy has to be measured on the specific vocabulary that carries clinical weight, not on overall word accuracy.
The natural challenge is whether a self-hosted system can match a managed cloud service under these constraints. To make the question concrete, we set a numeric bar: AWS Transcribe, the general managed baseline, reaches 0.82 medical-term recall on our simulated consultations. That is the target.
One clarification on scope. AWS offers a medical-specialised transcription product, but it supports English only and cannot process German. For our language the only available cloud option is the general service, and the general service is the 0.82 baseline.
Average word error rate is the not the primary metric here. A model can achieve a very low error rate on fluent conversational German and still miss most drug and brand names, because those terms are a small fraction of the word count and the entire value of the product. We therefore score recall on a curated set of medical terms.
The evaluation uses two datasets.
A synthetic capacity map: 86 curated German medical terms spanning drug ingredients, brand names, abbreviations, diagnoses, anatomy, and laboratory values, each rendered by four synthetic voices in four contexts (isolated, and inside three natural carrier sentences). This yields 1,376 clips per model, and across four candidate models we scored 5,504 transcriptions, plus a set of 12 dense utterances that pack several difficult terms into one sentence.
A real-world anchor: five synthetic (based on real world) complete doctor–patient consultations covering 27 gold terms, including a medication-heavy case and one in regional dialect.
Term matching normalises spelling and accepts known variants, so a standard abbreviation counts for its full form, while a phonetically wrong rendering does not. A single-phoneme error counts as a miss, because that is precisely what breaks downstream extraction and coding.
The synthetic set drives the model, category, and steering findings at scale. The five real consultations confirm that the synthetic findings hold on genuine speech.
The first design choice is counterintuitive. We began with a German-specialised speech model, on the reasonable assumption that a model tuned for German would be the best choice for German medical audio. It was the weakest option, and the least improvable.
Medical-term recall on the real anchor, single model, no additional processing:
The German-tuned model finished last by 8 to 10 points on the vocabulary the product depends on. It also carried a hidden failure: on the medication-heavy consultation it produced roughly half the words the general model did, silently dropping the first half of the conversation under identical settings. It is genuinely the strongest model on clean, simple speech and preserves dialect well, but a scribe that is fluent on easy input and drops content on hard input has optimised against the clinic.
Two alternative architectures were also evaluated and set aside:
The decisive factor was not the starting score. It was steerability. The entire strategy depends on biasing the model toward per-patient terms, and the German-tuned Whisper models cannot be biased this way without collapsing (Section 5). The correct base model is the one that can be improved, not the one with the best cold number. Moving to the general Whisper model raised the single-model recall from 0.65 to 0.73, still below the 0.82 cloud baseline. The remaining gap is closed by the pipeline.
Errors are not uniform, and knowing their structure is what makes the later layers targeted rather than speculative. Detection rate by category, for the specialised starting model versus the general model we adopted:
Brand names are the weakest category on every model, with drug ingredients close behind. Diagnoses, laboratory values, and anatomy are reliable on any competent model. A small, stable core of terms was missed by every model tested, including the cloud service: mostly anticoagulants, antidiabetics, and their brand names.
Two properties of the error distribution shaped the remaining architecture. Surrounding context recovers terms that fail in isolation, which argues for correcting at the sentence level rather than the word level. And most errors are near-misses rather than nonsense, which means a conservative spelling-level correction can recover a large share of them.
This layer produces the largest single improvement, and it is the one most implementations get wrong first.
General decode-parameter tuning was evaluated first and discarded: sweeping the usual settings recovered between zero and seven percent of missed terms.
The effective mechanism is contextual biasing, in which the model is nudged toward a supplied set of expected terms during decoding. Providing the target term directly recovered 96 percent of otherwise-missed terms on the general model. Two constraints govern how it can be used.
First, the base model must support it. The German-tuned Whisper models collapse when biased, degenerating to empty or repeated output, with recall falling from 0.65 to zero. This is the concrete reason base-model selection (Section 3) is dominated by steerability rather than baseline accuracy.
Second, the term set must be specific to the encounter, not a general dictionary. a broad medical dictionary regressed real conversations, because the model began inserting supplied terms that were never spoken. An earlier experiment injecting a general term list as decoding context showed the same effect, with recall falling as the list grew.
What works is a small, per-encounter term set: the medications the current patient is actually taking, assembled per visit. With that, both general models reach roughly 0.85 on real audio, and the difference between model sizes largely disappears. The general principle is that steering context should be narrowed to what a specific encounter will plausibly contain, not broadened to everything the domain might include.
A second, distinct failure mode is fabrication. Speech models can emit fluent, plausible text from silence, background noise, or a cough, because the underlying model always produces language. Typical artifacts include caption-style credits, filler phrases, applause, and stage directions. In consumer captioning this is harmless. In a clinical note it is a fabricated statement in a legal record.
Before any transcript reaches later stages, it passes through an output-integrity layer that removes these artifacts. In general terms, it filters known fabrication patterns, detects and removes repetition loops, rejects output in scripts that do not belong in the target language, screens for fluent-but-meaningless content, and masks profanity while preserving legitimate clinical vocabulary.
This layer does not improve recall and is not intended to. Its purpose is to ensure that a low-signal segment produces nothing rather than a confident fabrication. In a clinical scribe, an empty result is safe and an invented sentence is not.
Contextual steering requires terms known in advance. Consultations also produce terms that cannot be predicted: a newly prescribed drug, an unfamiliar brand, a medication with no prior record for the patient. A final correction layer addresses these.
The layer uses a small local language model constrained to a narrow task: it proposes only corrections of misheard terms toward recognised spellings, and a deterministic rules stage decides what is applied. Corrections are accepted only when the change is a small spelling adjustment toward a plausible medical term. Changes to numbers, doses, and units are never permitted, and the layer cannot rewrite meaning or introduce content. An early design that asked a single model to both correct and extract in one pass proved unreliable on exactly the difficult, term-dense cases, and it added significant latency for no gain. Narrowing the task to spelling corrections alone, under strict rules, made it both fast and safe.
The governing principle is to ask the model for the difference, not the document. Requesting a full rewrite is where a small model truncates, fabricates, or alters meaning; requesting a short list of conservative corrections is not.
The complete pipeline was scored against the cloud baseline on identical audio with the identical matcher.
The cloud service beats a single self-hosted model. The self-hosted pipeline beats the cloud service, 0.91 to 0.82, by recovering the hard core during correction. Adding the contextual steering to the underlying model widens the margin further.
The result is not an artifact of the five-sample anchor. On the synthetic corpus the cloud service lands where every other engine does, at 0.57 overall recall, with brand names the weakest category at 0.25 and the same anticoagulant core missing. On the dense multi-term utterances it reached 0.58 drug recall and 0.64 term recall. The cloud model has the same failure profile the self-hosted pipeline was built to correct, at synthetic scale as well as on real audio.
The common framing, that the cloud is the ceiling and self-hosting is the compromise, is therefore inverted for non-English medical speech: the cloud beats a single model, and a layered pipeline beats the cloud. Three points hold regardless of the exact numbers. The medical-specialised cloud product does not support German, so the comparison is against the general service. The service is cloud-based and the system is on-premise, so recall parity would not restore data control, and here the pipeline exceeds parity. And the cloud service misses the same terms, so a correction layer would be required on top of it as well.
The broader methodological point is to compare a pipeline against a pipeline. A full system benchmarked against a single external component is not a like-for-like comparison.
Throughput and latency are not constraints for this design. On a single modern accelerator the candidate models run at roughly 20 to 40 times real-time, and the correction stage adds a small fixed latency to each finalised transcript. Model choice is therefore a quality decision rather than a latency one. The system runs on hardware already owned, with no per-minute usage cost and no protected health information leaving the premises.
The pipeline is a sequence of narrow stages, each responsible for one transformation:
The design principles behind it:
For context, the system began as a single general speech model feeding a single note-generation prompt in batch. The evolution was toward smaller, specialised stages: streaming instead of batch, per-encounter instead of generic context, and one responsibility per stage instead of one model doing everything. The approach generalises to any language where the clinically important vocabulary is underrepresented in general training data.
For non-English medical transcription, the practical conclusion is to compare pipelines rather than components, and to build the layers that address a known failure profile. On our evaluation the self-hosted pipeline exceeded a leading cloud service in German, on owned hardware, and the design transfers to other languages where the words that matter most are the ones general models were least trained to hear. Optimizing Self-Hosted Whisper on German Medical Speech was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.