![AI Medical Alerts: 90% Ignored at Vanderbilt [2026]](https://shattered.io/wp-content/uploads/2026/10/ai-medical-alert-fatigue-chatbots-real-world-test-2026-photo.webp)
Medical AI now faces its hardest exam in the one place demos never reach, the clinic message queue. A story from the Association of Health Care Journalists’ Health Beat, titled “Alert fatigue, chatbots: the real-world test for AI medical technology” and attributed to Farah Yousry, puts two failure modes side by side. False-positive AI alerts teach clinicians to look away. Patient chatbots overwhelm clinic inboxes.
Researchers have been saying for a while that benchmark wins are not enough. Michael Rodman, a researcher studying medical AI, explained the logic of testing a system on real emergency-room visits instead of curated cases in IEEE Spectrum: “We went the extra step and showed that this performance also works in the real world.” That sentence sums up the standard the AHCJ story applies to the whole field as of October 27, 2026.
The gap between a pilot and a busy floor is not new, and it is not limited to one country. We saw it in a UAE medical AI pilot built around data residency, where the hard questions were about workflow and governance rather than raw model accuracy. This analysis walks through the numbers behind alert fatigue, the chatbot inbox problem, the newest medical models shipping this autumn, and what each group in the chain should do next.
Don't miss new tech stories on Google
Add Shattered once in the Google app and our stories show up in your news suggestions.
What the AHCJ Health Beat Story Reports #
Health Beat is the publication of the Association of Health Care Journalists, a group that trains reporters who cover medicine. The story carries the byline of Farah Yousry, whose author page on the AHCJ site lists it among her recent work. Its search-result summary says experts discussed false-positive AI alerts and patient chatbots overwhelming clinic inboxes at the AHCJ’s fall summit.
That framing matters because it moves the debate away from model scores. Most public conversation about medical AI still centers on accuracy tables, exam results, and head-to-head comparisons with doctors. The AHCJ story asks a different question. What happens to a hospital when the software is right often enough to be adopted, yet wrong often enough to bury staff?
A transparency note on scope. The full text of the Health Beat piece is the primary document, and this analysis relies on the headline, byline, summary, and the confirmed details from related coverage of the same themes. Where a detail is not confirmed, this article says so instead of filling the gap. Every number below comes from a named outlet or institution, and reasoning that belongs to this publication is labeled as analysis.
Two threads run through the story. One concerns clinicians, who receive automated warnings inside electronic health record systems. The other concerns patients, who now have chatbots that make it cheap to send a message. Both threads end at the same bottleneck, a finite number of human attention hours in a clinic.
Vanderbilt’s Alert Data: Up to 90% Ignored #
The firmest numbers in this story come from Vanderbilt University Medical Center. Researchers there reported that up to 90% of computerized clinical alerts were ignored, a pattern that feeds what the field calls alert fatigue. They analyzed two years of alert-user-interaction data covering nearly 3 million alerts, then tested whether AI could suggest better alert settings.
The AI produced 1,727 alert-optimization suggestions for the team to review. Of those, 76 matched later manual alert updates, meaning clinicians eventually made the same change by hand. Another 20 aligned with clinician-determined best practices. The researchers calculated that those 96 recommendations could have eliminated 9.3% of the alerts analyzed while maintaining patient safety. The project involved a researcher identified as Liu in the available summary.
| Metric | Reported figure | How to read it |
|---|---|---|
| Computerized clinical alerts ignored | Up to 90% | Core alert fatigue signal |
| Data window | Two years | Long enough to capture seasonal workload shifts |
| Alerts analyzed | Nearly 3 million | Large real-world sample, one health system |
| AI-generated suggestions reviewed | 1,727 | The raw pool of machine proposals |
| Matched later manual updates | 76 | AI anticipated changes staff made anyway |
| Aligned with clinician best practices | 20 | Independent check against expert judgment |
| Combined useful recommendations | 96 | About 5.6% of the pool (this publication’s arithmetic) |
| Potential alert reduction | 9.3% | Estimated with patient safety maintained |
Read the table with care. The headline 90% figure is an upper bound (“up to”), not an average across every alert type. The 5.6% share is simple division of 96 by 1,727, and the study summary does not describe what happened to the other 1,631 suggestions. They may have been neutral, redundant, unsafe, or simply unreviewed. Treat them as unknown.
Still, the result cuts in two directions. On the hopeful side, an AI system proposed changes that clinicians later made on their own, which suggests it can find real noise. On the sobering side, finding 96 useful ideas in a pool of 1,727 means a human still has to sift roughly 18 suggestions to land one that holds up. The tool that fights alert fatigue creates a review queue of its own.
Why Alert Fatigue Is an AI Problem Too #
Alert fatigue predates large language models. Computerized warnings about drug interactions, duplicate orders, and abnormal results have long shaped clinician habits in electronic health records. What changes with AI is the supply of new alerts. A radiology model, a sepsis predictor, and a suicide-risk flag each add their own stream to a system where up to 90% of the existing alerts already get ignored.
Here is the analytical core. Every false positive spends a little of the clinician’s trust, and trust is a depletable resource. A model with strong overall accuracy can still produce more false alarms than true ones when the condition it hunts for is rare. In that regime, the busiest users learn that the alert is usually wrong and start clicking past it. The rare true positive then gets the same treatment.
The AHCJ story names false positives as the trigger. That is the right target. Teams that judge their products on sensitivity alone will keep shipping tools that look excellent in a validation report and exhausting on a ward. A better yardstick tracks how often clinicians act on an alert, how long they spend per alert, and what share they override without reading.
There is also a measurement trap. If a hospital counts alerts shown instead of alerts acted upon, a noisy model can look like a diligent one. Vanderbilt’s decision to study user interaction data, rather than model output alone, is the template worth copying. It measures the human end of the pipe.
Graham Walker’s Headache Exercise #
The setup
Graham Walker, M.D., co-founder of MDCalc, led an interactive exercise at an AHCJ event built around a patient with “the worst headache of her life.” The case contained a deliberate conflict. An initial CT scan was described as having 98.7% accuracy for testing negative for a brain bleed, yet an AI radiology system flagged possible bleeding.
Participants used ChatGPT to help decide the next step. The reported responses varied widely, from ordering additional testing, to discharging the patient without further testing, to asking the doctor in charge. The point of the exercise was not to crown a correct answer. It showed how two trusted signals can disagree and how a chatbot slots into that disagreement.
What the split answers show
Three lessons follow, and they are this publication’s analysis rather than reported findings. First, a single chatbot can lead different people to opposite conclusions on the same facts, depending on phrasing and context. That is a reproducibility problem, and it matters most when the stakes are highest. Second, the exercise exposes how a flagged scan and a reassuring scan force a decision under uncertainty, which is exactly the moment people reach for a quick answer.
Third, the range of outcomes included “ask the doctor in charge” as one valid path. That option is the safest design pattern in the room. A tool that routes an ambiguous case upward costs a few minutes. A tool that nudges a clinician toward discharge when a scan is contested can cost far more. If you want a primer on how the leading assistants differ in practice, our comparison of Claude, Gemini, and ChatGPT shows why the same prompt does not produce the same answer across models.
Patient Chatbots and the Overloaded Clinic Inbox #
The second half of the AHCJ headline concerns chatbots, and the confirmed claim is narrow. Patient chatbots can overwhelm clinic inboxes. The story summary does not publish message counts, so no volume figure appears here. The mechanism, however, is easy to reason about.
Sending a message used to take effort. A patient had to call, wait on hold, or compose a portal note. A chatbot removes that friction and can draft a polished message in seconds. When the cost of sending drops, volume rises, while the number of nurses and physicians who read the inbox stays flat. Queueing theory predicts the result: longer waits, buried urgent items, and staff who triage by skimming.
Vanderbilt Health News describes work on both sides of this loop. Its roundup covers AI that helps patients craft better portal messages and reports that ChatGPT-3.5 and ChatGPT-4 outperformed doctors in answering patient questions submitted through patient portals. Better messages and better answers sound like pure wins. Yet they also make it easier to generate more traffic, and the clinic still has to read it.
There is a quality dimension too. A well-written message from a chatbot can hide a vague or inaccurate clinical claim behind confident prose. A nurse who once spotted worry in a halting, two-line note now reads an organized paragraph that sounds calm. The signal staff relied on, the patient’s own wording, gets smoothed away.
The same dynamic has appeared outside medicine. When six AI agents failed a £350 train refund test, the lesson was that fluent systems still stumble on multi-step real-world tasks. Healthcare adds a harsher twist. A failed refund costs money, and a failed triage costs far more.
What Researchers and Advisers Say About Health Chatbots #
Several named experts have weighed in on chatbot safety in recent months. Their remarks come from separate outlets and address consumer use, but they bear directly on the clinic-inbox problem, since patients bring chatbot-shaped advice with them.
Ome Ogbru, identified as CEO of AI Engines, warned in a NewsNation report: “You should never skip meeting with your health care professional.” In the same context he added: “You should not treat these platforms as giving advice.” Both lines frame chatbots as aids and not as replacements for a licensed clinician.
Nicholas Tiller, a researcher who evaluated health-chatbot responses, was more blunt in The Philadelphia Inquirer. Assessing dangerous incorrect answers he found while testing multiple chatbots with medical questions, he said: “It would more than likely cause somebody harm if they were to follow the advice.” He also summarized the gap between fluency and safety: “Chatbots are not designed for health.”
Put these voices next to the AHCJ story and a pattern emerges. Consumers get chatbot answers that can be wrong, and some of them forward those answers to clinics. Clinics then spend scarce hours correcting, reassuring, or re-triaging. Rodman’s real-world standard, quoted at the top, is the countermeasure. Systems that only look good on benchmarks should not reach patients or inboxes.
Automation Bias and Deskilling: The Vanderbilt Review #
A 2026 review from the Department of Biomedical Informatics at Vanderbilt University Medical Center takes the argument a step further. Titled “Beyond alert fatigue: Automation bias, deskilling, and clinical decision-making in the age of AI,” it lists Scott D. Nelson and Adam Wright as authors and went online ahead of print on July 30, 2026, in AJHP.
Per the abstract text available in search results, the authors compare AI to autopilot in aviation. Autopilot improves flight safety, and AI could likewise improve patient safety and the medication-use process. But the risks come not from malfunction alone. They come from how humans interact with automation, especially automation bias, the tendency to over-rely on a machine’s suggestion. The review also flags deskilling, the gradual erosion of manual or cognitive abilities.
The title does quiet work. By saying “beyond alert fatigue,” the authors treat noise as the first problem and not the last. Once clinicians trust the tool, a different failure appears. They stop checking. In the headache exercise, that is the clinician who accepts a flagged scan or a clean scan without a second look because a machine agreed with one of them.
Alert fatigue and automation bias are opposite errors. One ignores the machine too often, the other obeys it too often. A safe system has to manage both, and the same dashboard cannot fix them. Override rates catch the first. Audit samples of accepted recommendations catch the second.
Vanderbilt’s Other Tests: Suicide-Risk Flags and V-CARES #
Vanderbilt Health News lists several other efforts that bear on the real-world test. At three neurology clinics, a Vanderbilt AI system flagged 8% of arriving patients as having a relatively high risk of a suicide attempt within the next 30 days. The goal, according to the roundup, is to prompt conversations, not to replace judgment.
Do the arithmetic on an alert rate like that. Eight flags per 100 arrivals is a steady stream in any busy clinic. If most flags prove unfounded, fatigue sets in quickly, and the one that matters risks being skipped. If the flag opens a short, scripted conversation, the workflow absorbs the volume. Design determines which outcome occurs, not the model alone.
The second effort targets chatbot quality head on. According to Vanderbilt Health News, the university received up to $7.3 million from the Advanced Research Projects Agency for Health to build the Vanderbilt Chatbot Accuracy and Reliability Evaluation System, or V-CARES. The system is meant to evaluate hallucinations, omissions, and misaligned values in AI-generated responses, with mental-health communication as the first demonstration area.
V-CARES matters for the same reason the Vanderbilt alert study does. It measures behavior against clinical judgment rather than against a leaderboard. Omissions deserve special attention. A chatbot answer can be true in every sentence and still dangerous because it leaves out the caution that a clinician would have added.
New Medical AI Models Raise the Stakes #
The real-world test is arriving just as the newest models reach clinicians. Reports from AI Weekly and TechTarget, citing STAT, say OpenEvidence launched medical AI models on September 3, 2026. Three production models serve verified clinicians. Osler returns answers in about five seconds, Sackett takes about 30 seconds for deeper searches, and Snow runs about five minutes for literature review.
A fourth model, Darwin, is positioned as a research partner. OpenEvidence says it scored 100% on the MedQA benchmark, though that is a company claim and independent replication was not part of the reports. Access is restricted over dual-use concerns in areas such as virology and genetics. Coverage differs on whether to count Darwin in the launch tally, which explains why some headlines say three models and others say four.
| Development | Date reported | Detail | Caveat |
|---|---|---|---|
| OpenEvidence Osler, Sackett, Snow | Sept. 3, 2026 | Responses in about 5 seconds, 30 seconds, and 5 minutes | Speeds are reported figures, not independent tests |
| OpenEvidence Darwin | Sept. 2026 reports | Research-partner model, 100% on MedQA per the company | Access restricted, claim not independently verified |
| Northwestern Medicine and OpenEvidence | From Oct. 8 | Single sign-on access inside the Epic EHR, per TekinHealth | No outcome data yet |
| Cigna and OpenAI reasoning models | Sept. 26, 2026 report | Tools for oncology nurses and case managers, per MedSpark | Cigna has not said which models or how answers will be checked |
| NVIDIA healthcare partners | Oct. 21, 2026 | Avelios, Corti, Dedalus, Dawn Health, Sword build on Nemotron | Product claims come from the vendor announcement |
Two details deserve a close read. Northwestern’s Epic integration puts a clinical search tool one click from the chart, which is where alert fatigue already lives. Cigna’s oncology tools show a payer moving AI toward nurses and case managers, who coordinate care for members with complex illness. Neither rollout has published outcome data in the reports reviewed here.
Speed claims also deserve skepticism. A five-second answer is a feature in a quiet moment and a hazard when a tired clinician accepts it unchecked. The faster the tool, the more the surrounding workflow must slow down the decisions that carry risk.
Four Ways to Manage Clinical AI Noise Compared #
Health systems have four broad options for handling the noise problem, and each trades one risk for another. The comparison below is this publication’s framework, populated with the confirmed figures from the stories above.
| Approach | Strength | Weakness | Evidence from this story |
|---|---|---|---|
| Static rule-based alerts | Predictable and easy to audit | Alerts pile up and get ignored | Up to 90% ignored at Vanderbilt |
| AI-tuned alert settings | Finds noise humans missed | Needs expert review of every suggestion | 96 useful of 1,727, 9.3% potential cut |
| Clinician-facing LLM search | Fast answers inside the chart | Automation bias and deskilling | Vanderbilt review, Oct. 8 Epic integration |
| Patient-facing chatbots | Easier access for patients | Inbox overload and unsafe answers | AHCJ story, Tiller and Ogbru warnings |
No row wins outright. The first approach is stable but noisy, and the second reduces noise slowly. The third speeds up clinicians while inviting over-trust. The fourth widens access while raising volume. Mature programs will mix all four and measure each with its own metric.
One practical consequence follows. The second approach, AI tuning of alert rules, is the least glamorous and arguably the most defensible. It targets a documented harm, uses a clear safety constraint, and keeps clinicians in charge of the final change. The Vanderbilt numbers are modest, but they are real and measured against human judgment.
Historical Context: From Pop-Up Warnings to Chatbots #
Hospitals adopted computerized alerts to catch errors people miss, and for many cases they work. Over the years, each new safety concern spawned another rule, and each rule spawned another pop-up. Nobody owned the total. The result is the pattern Vanderbilt measured, a system where most warnings are dismissed.
The first wave of medical machine learning repeated the pattern at a larger scale. Predictive models for imaging triage and risk scoring produce scores that need a threshold, and every threshold trades false positives against missed cases. A low threshold avoids misses but feeds the noise, which is the tension the AHCJ story describes.
Generative chatbots opened a third front. Where alerts push information toward clinicians, chatbots pull questions toward the clinic from outside. Patients arrive with drafted messages, interpreted lab values, and confident summaries. The old problem of too many pop-ups became the new problem of too many well-written messages.
Each wave added capability and added load. The AHCJ headline is a reminder that capability alone never settled the matter. What settled earlier technology was workflow design, ownership of the alert catalog, and honest measurement. Those lessons carry over directly.
Market Impact for Health Systems, Vendors, and Developers #
Health systems
Buyers will start asking for workflow evidence before they sign. A vendor that shows override rates, time per alert, and message-volume effects will beat one that shows only a benchmark chart. Health systems that already log alert interactions, as Vanderbilt did across two years, hold a data advantage for judging new tools.
Security and privacy also shape the decision. Clinical AI touches records that regulators guard closely, and breaches in healthcare keep the compliance clock in view, as our report on the Quest breach and the 60-day HIPAA window showed. A tool that adds workload and risk at once is a hard sell.
Vendors
The OpenEvidence and NVIDIA-partner announcements show how crowded the field has become in a single autumn. Differentiation will shift from who has the highest benchmark score to who creates the least extra work. Restricted-access releases, like the one reported for Darwin, signal that vendors now treat some capabilities as too risky for open distribution, and that stance will shape procurement conversations.
Developers
Engineers building on these models inherit the alert-fatigue problem the moment they add a notification. Ship fewer, better prompts to humans. Log what users do with each one. Treat an ignored alert as a bug report. The wider AI and machine learning coverage on this site tracks the model releases that make these choices urgent.
Five Predictions for Medical AI Over the Next Year #
These are forecasts from this publication, not reported facts, and each could prove wrong.
- Override rate becomes a sales metric. Vendors will publish how often clinicians accept or dismiss their alerts, because buyers have seen the Vanderbilt-style numbers and will ask.
- Clinics cap and tier chatbot traffic. Expect structured intake forms, message limits, and automatic sorting of urgent from routine items, since an open inbox cannot absorb unlimited fluent text.
- Evaluation moves to the workflow. Procurement checklists will add time-to-triage, alert burden per clinician, and error recovery to accuracy scores, following the real-world standard Rodman describes.
- Restricted-access models spread. Darwin’s limited release will serve as a template for dual-use medical capabilities, with verified-user gates and audit trails.
- Disagreement drills enter training. Exercises like Walker’s headache case, where a scan and an AI flag conflict, will become standard teaching for both residents and journalists covering the field.
The common thread is measurement. Each prediction assumes that organizations start counting what humans do with AI output, not only what the AI outputs.
What AI Teams Should Test Before Shipping to a Clinic #
Teams building medical assistants can borrow a short checklist from the stories above. It does not require a hospital partnership to start. A scoring harness that runs the same medical prompts across models and tracks variance would have flagged the split answers in Walker’s exercise early. Our walkthrough of a chatbot scoring harness shows how to build that kind of repeatable test.
- Run each clinical prompt several times and record how often the answers disagree.
- Test omissions separately from errors, since a true but incomplete answer can still mislead.
- Measure alert burden per user, and set a budget before launch.
- Log overrides, accepts, and time spent, then review a sample of accepted outputs to catch automation bias.
- Give every ambiguous case a clear route to a human, mirroring the “ask the doctor in charge” option.
None of these steps requires new science. They require the discipline to treat the clinic as the test environment, which is the point of the AHCJ headline. The takeaway: the field will be judged on attention saved or lost, not on demo accuracy.
FAQ: Alert Fatigue and Medical Chatbots #
What is AI alert fatigue?
Alert fatigue is the habit of ignoring warnings after seeing too many of them, especially false positives. A Vanderbilt University Medical Center study reported that up to 90% of computerized clinical alerts were ignored. AI can worsen the problem by adding more alert streams, or it can help by tuning existing ones.
Who wrote the story behind this analysis?
The headline is “Alert fatigue, chatbots: the real-world test for AI medical technology.” It ran on the Association of Health Care Journalists’ Health Beat and is attributed to Farah Yousry. Its summary says experts discussed the issues at the AHCJ’s fall summit.
How many AI alert suggestions did Vanderbilt find useful?
Of 1,727 AI-generated suggestions, 76 matched later manual alert updates and 20 aligned with clinician-determined best practices. That totals 96. The researchers calculated those 96 recommendations could have eliminated 9.3% of the alerts analyzed while maintaining patient safety.
What was the headache exercise?
Graham Walker, M.D., co-founder of MDCalc, led an exercise at an AHCJ event about a patient with “the worst headache of her life.” A CT scan described as 98.7% accurate for testing negative for a brain bleed conflicted with an AI radiology flag of possible bleeding. Participants used ChatGPT, and their responses ranged from more testing to discharge to asking the doctor in charge.
Can patients rely on chatbots for medical advice?
Experts quoted in recent coverage say no. Ome Ogbru advised that people should never skip meeting with a health care professional, and researcher Nicholas Tiller said chatbots are not designed for health. Patients can use chatbots to prepare questions, but a clinician should confirm anything that affects care.
Which new medical AI models launched recently?
Reports say OpenEvidence launched Osler, Sackett, and Snow for verified clinicians on September 3, 2026, plus a restricted research model called Darwin. Cigna and several NVIDIA partners also announced healthcare AI work this autumn. Independent outcome data for these tools had not appeared in the reports reviewed.
What should hospitals measure when adopting medical AI?
Track how often clinicians act on or override each alert, how much time each one takes, and how message volume changes after a chatbot launches. Also sample accepted AI outputs to check for automation bias. Accuracy alone does not capture the burden on staff.