Medical AI was meant to help. This week it replaced nurses and dodged its own checks

Nurses at Montefiore hospital in New York were laid off and replaced by AI software for utilization review, according to their union. Separately, a former Mayo Clinic leader filed a lawsuit alleging the hospital demoted and fired her for raising safety concerns about its AI assistant MAYA, which she claims had error rates as high as 67%. The cases highlight risks of medical AI replacing jobs and bypassing oversight.

The pitch for medical AI is that it frees clinicians to care for patients. Two stories this week suggest the reality can run the other way. In New York, nurses say software replaced them. In Minnesota, a former Mayo Clinic leader says the software was not safe to trust. Marilyn Shuler spent 39 years reading patient charts at Montefiore hospital in the Bronx. This month she was one of a dozen nurses laid off, replaced by AI software that does part of the same job, their union says. The nurses handled utilization review. They read charts and argue with insurers over what care gets covered. The New York State Nurses Association says the layoffs broke a contract the nurses had just won through a strike, the Guardian reported https://www.theguardian.com/technology/2026/jul/13/nurses-new-york-ai . “It should also concern every practitioner and patient who cares about… the quality of care they receive,” said Shaiju Kalathil, a Montefiore nurse and union representative. Montefiore disputes the account. It calls the union’s claims “inaccurate and misleading”, and says the change involves a nonclinical, paperwork programme. What the whistleblower saw If New York is about jobs, Minnesota is about trust. And it is more alarming. Traci Tamiko Eto joined the Mayo Clinic in 2023 to help build safeguards around its AI. In a new lawsuit, she says the hospital demoted her and then fired her for raising the alarm about its AI rollout, Minnesota Public Radio reported https://www.mprnews.org/story/2026/07/09/lawsuit-alleges-mayo-clinic-cut-corners-with-ai . Her claims are specific. The team behind an AI assistant called MAYA, she alleges, deleted unflattering test results, overstated what the tool could do, and pushed it forward without proper oversight. At one point, the suit says, the tool had an error rate as high as 67%. Mayo says it is committed to responsible AI, with privacy, security, and compliance built into its work. Eto’s account, if it holds up in court, describes something else. It describes a system rushed past the checks meant to catch its mistakes. The gap between promise and practice These are two cases, not a trend. But they sit either side of the same fault line. Healthcare is where AI’s benefits are meant to be clearest, and where its failures are least forgivable. A mangled advert is embarrassing. A mangled chart is dangerous. The wider record is already uneasy. Utah has let an AI renew prescriptions https://thenextweb.com/news/utah-ai-prescription-refills-doctronic without a doctor. Research keeps finding that leaning on AI can quietly erode the skills https://thenextweb.com/news/ai-never-skilling-critical-thinking-research professionals are meant to keep, and that it is cutting experienced careers short https://thenextweb.com/news/ai-older-workers-careers-cut-short-research . Employers from Allianz https://thenextweb.com/news/allianz-1800-job-cuts-ai to Thomson Reuters https://thenextweb.com/news/thomson-reuters-engineering-layoffs-ai are already swapping staff for models. Who carries the risk The common thread is who ends up holding the risk. Montefiore keeps the savings; the patient inherits whatever the software misses. Mayo gets the efficiency; the whistleblower, she says, got shown the door. Neither case has been tested in court, and both hospitals defend their record. But together they puncture the tidy story that medical AI only ever adds a helping hand. Sometimes it removes one. And sometimes the person who understood the machine best is the first to be told to leave. Get the TNW newsletter Get the most important tech news in your inbox each week.