A 58-year-old patient dials their primary care clinic at 8:42 AM.
The audio stream captures a tremor in the caller’s voice. Their breathing is shallow and rapid. Between s, they state:
“I’ve had this crushing pressure in the center of my chest for the last twenty minutes… it’s radiating down my left arm and I feel nauseous. Do you have an opening to see Dr. Miller today?”
In a clinic with a human receptionist, immediate protocol takes over: the receptionist interrupts the patient, asserts clinical priority, hits the emergency trunk line, or initiates a warm handoff to a triage nurse while instructing the caller to stay on the line or dial 911 immediately.
In a modern enterprise deploying an unconstrained, conversational voice agent — built on top of standard WebRTC/SIP streaming, an off-the-shelf Automatic Speech Recognition (ASR) pipeline, and an LLM orchestration loop — an entirely different sequence unfolds.
The speech engine converts the incoming audio to text. The agent’s prompt instructions state:
You are a warm, empathetic front-desk receptionist for Miller Family Medicine.Your goal is to assist patients with booking appointments, answering questions, and collecting intake information. Always maintain a polite, professional tone.If a patient mentions a life-threatening emergency, instruct them to hang up and call 911 immediately.
The model ingests the caller’s utterance. However, the user’s input contains two competing intents:
Because language models operate on probabilistic sequence completion rather than strict hierarchical constraint satisfaction, the conversational agent fixes on the explicit transactional task. It queries the electronic health record (EHR) scheduling API via a tool call:
{ "tool": "check_provider_availability", "parameters": { "provider_id": "dr_miller_409", "requested_date": "2026-09-30" }}
The tool returns zero same-day slots. The next available appointment is the following Tuesday at 10:30 AM.
The model parses the tool response, synthesizes its conversational output, and streams synthetic voice audio back over the SIP trunk:
“I am so sorry to hear you aren’t feeling well! Dr. Miller is completely booked for today, but I can get you in first thing next Tuesday at 10:30 AM. Would you like me to reserve that time for you?”
Twenty minutes later, the patient suffers complete cardiac arrest at home.
Why does an LLM fail at basic clinical triaging even when provided explicit system prompts?
THE UNBOUNDED INTAKE LOOP (FAILURE ARCHITECTURE):┌────────────────────────┐ ┌───────────────────────────┐ ┌───────────────────────────────┐│ Inbound Telephony SIP ├─────►│ ASR & Transcription ├─────►│ Generative LLM Planning Loop ││ (Patient in Distress) │ │ Audio to Text Stream │ │ (System Prompt Guardrails) │└────────────────────────┘ └───────────────────────────┘ └─────────────┬─────────────────┘ │ ▼ Evaluates both Intents: │ - "Chest pain" (Symptom) │ - "Book visit" (Action) ▼ ┌───────────────────────────────┐ │ Model Prioritizes Task Loop: │ │ Queries EHR Scheduling API │ └─────────────┬─────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Books Slot for "Next Tuesday" │ │ FATAL CLINICAL DRIFT │ └───────────────────────────────┘
The breakdown stems from three fundamental flaws in standard conversational AI architecture:
Generative models are fine-tuned to be helpful assistants. When a user presents an operational goal (“Can I see the doctor today?”) wrapped in contextual detail (“my chest hurts”), the model prioritizes fulfilling the request. Under multi-turn conversation, prompt-based constraints experience semantic dilution: conversational momentum overrides negative constraints.
Patients in acute distress do not use clean, clinical keywords. They rarely say: “I am experiencing symptoms consistent with an acute myocardial infarction.”
They say:
Probabilistic attention heads often fail to categorize subtle, indirect expressions of life-threatening decompensation as emergencies when weighed against a direct scheduling query.
In a standard streaming voice pipeline, token generation is tied to text-to-speech (TTS) buffers. If an emergency phrase is recognized mid-turn, an un-governed agent cannot sever the connection at the transport layer without waiting for the current generation buffer to flush.
Clinical safety cannot be outsourced to a system prompt. In healthcare operations, triage is not a conversational topic — it is a system invariant.
At Claire, we separate conversational processing from clinical safety by placing a Deterministic Triage Interceptor directly in the audio transport pipeline, running out-of-band and ahead of the generative model.
CLAIRE DETERMINISTIC TRIAGE ARCHITECTURE:┌────────────────────────┐│ Inbound Telephony SIP │└───────────┬────────────┘ │ ├─── Raw Audio Stream (RTP) │ ▼┌────────────────────────────────────────────────────────────────────────────────────────┐│ DETERMINISTIC TRIAGE RUNTIME INTERCEPTOR (Sub-800ms Pipeline) ││ ││ [Pipeline Layer 1: Acoustic Stress & Biomarker Telemetry] ││ - Real-time jitter, pitch tremor, respiratory gasping detection ││ ││ [Pipeline Layer 2: Deterministic Aho-Corasick Keyword Automaton] ││ - Sub-millisecond matching against emergency taxonomy: ││ {"crushing chest", "left arm", "shortness of breath", "anaphylaxis", ...} ││ ││ [Pipeline Layer 3: Synchronous Fast-Classifier (Deterministic AST)] ││ - Zero generative completion; binary classification only: ││ IsEmergency(input) -> TRUE / FALSE │└───────────────────────────┬────────────────────────────────────────────────────────────┘ │ ┌─────────────┴─────────────┐ │ │ ▼ (PASS: Normal Intake) ▼ (BREACH: Triage Alert Triggered)┌───────────────────────────┐ ┌────────────────────────────────────────────────────────┐│ Generative Inference Node │ │ HARDWARE INTERRUPT (SIP REFER / Cold Transfer) ││ - Intent Parsing │ │ 1. Sever Generative Context Window Immediately ││ - Appointment Booking │ │ 2. Execute SIP Transfer to 911 / Live Triage Nurse ││ - Zero Triage Authority │ │ 3. Inject Telemetry Packet to Clinic Emergency Queue │└───────────────────────────┘ └────────────────────────────────────────────────────────┘
The raw telephony stream (RTP) is split at the SIP gateway. Audio packets are transcribed by a streaming ASR engine that emits partial transcripts every 250 milliseconds. The transcript tokens do not flow directly into the LLM context; they hit the Invariant Gateway first.
Before any neural model evaluates semantic context, the partial transcript is evaluated by an exact-match automaton (such as an Aho-Corasick tree) populated with clinical red flags across cardiogenic, respiratory, neurological, and anaphylactic symptom sets.
from ahocorasick import Automatonclass TriageSafetyGateway: def __init__(self, emergency_lexicon: list[str]): self.automaton = Automaton() for idx, phrase in enumerate(emergency_lexicon): self.automaton.add_word(phrase.lower(), (idx, phrase)) self.automaton.make_automaton() def scan_partial_transcript(self, transcript_chunk: str) -> tuple[bool, str | None]: text = transcript_chunk.lower() for _, (_, matched_phrase) in self.automaton.iter(text): # Deterministic, non-probabilistic match return True, matched_phrase return False, None
If an invariant breach occurs:
The total elapsed time from the caller uttering a life-threatening phrase to telephony rerouting is under 800 milliseconds.
For engineering leaders designing digital labor for healthcare front doors:
When algorithms answer the phones in healthcare, failure is not a UI glitch. Build systems with invariant runtime boundaries.
The Anatomy of Clinical Triage Drift: Why Conversational AI Cannot Guard the Front Door was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.