Microsoft’s AI got 83% of medicine’s hardest cases right in a Science study, more than double the doctors’ score

Microsoft's MAI-DxO diagnostic system answered 83.2% of 304 complex New England Journal of Medicine cases correctly in a peer-reviewed Science paper published Thursday, versus 36.1% for generalist physicians allowed textbooks and the internet and 22.3% without them. The orchestrator, which runs on OpenAI's GPT-5 and has several AI agents debate diagnoses and weigh test costs, scored 80.3% on emergency department cases from MIMIC-IV-ED against 46.8% for physicians, though its most accurate setting averaged $5,431 in tests per case versus $4,724 for the doctors. Microsoft AI CEO Mustafa Suleyman frames the result as a step toward "medical superintelligence," while the authors caution the findings come from a controlled benchmark and the system has only been tested on past cases.

Mustafa Suleyman, now CEO of Microsoft AI, photographed in 2018 when he was at DeepMind in London. Image: Joi Ito <https://commons.wikimedia.org/wiki/File:Mustafa Suleyman 29099346447 .jpg / Wikimedia Commons, CC BY 2.0 https://creativecommons.org/licenses/by/2.0/ , cropped Microsoft says its AI diagnostic system got the right answer on 83% of medicine’s hardest published cases, more than double the score of doctors given the same cases. The results, for a system called MAI-DxO, appeared on Thursday in a peer-reviewed paper in Science https://www.science.org/doi/10.1126/science.aeh7930 , and Microsoft presents it as a step https://microsoft.ai/news/building-medical-superintelligence/ towards “medical superintelligence”, the goal its AI chief Mustafa Suleyman has set for the company. MAI-DxO, short for Microsoft AI Diagnostic Orchestrator, works like a virtual panel of doctors. Several AI agents each propose diagnoses, pick questions to ask or tests to order, and challenge the panel’s emerging view over several rounds of debate, weighing what each test is likely to reveal against what it costs. How the AI and the doctors were tested Microsoft built its own benchmark, SDBench, from 304 complex case reports published in The New England Journal of Medicine and a second set of more ordinary emergency department visits from MIMIC-IV-ED, a database of admissions at Beth Israel Deaconess Medical Center in Boston. Each case starts with a short description of the patient. The AI, or the doctor, then has to ask for each piece of information, from the patient’s history to a scan, and every test adds to an estimated bill. The paper reports these results: - Complex NEJM cases: MAI-DxO 83.2%; generalist physicians 36.1% with textbooks and the internet, 22.3% without them. - Emergency department cases: MAI-DxO 80.3%; physicians 46.8%. - Cost: on the NEJM cases its most accurate setting averaged $5,431 in tests per case, against $4,724 for the doctors using references. A setting capped at a $1,000 budget scored 79.6% at $1,303 per case. The doctors were primary care physicians and general internists working alone. Cases this rare would normally go to specialists or teams of doctors, and none were compared. The authors also note the limits of the set-up in the paper’s summary: “Although these findings were obtained in a controlled benchmark setting, they highlight the promise of language models for supporting more accurate, lower-cost diagnoses.” How much of it is GPT-5? The headline system runs on OpenAI’s GPT-5, and much of the gain comes from that model. Microsoft’s own figures show GPT-5 alone, with no orchestration, scored 78.9% on the NEJM cases at $8,409 per case. MAI-DxO added about four points and cut the bill by about $3,000. Other models on their own trailed: Grok 4 scored 76.6%, Claude 4.1 Opus 74.0% and Gemini 2.5 Pro 73.0%. The orchestrator helps most with weaker models and harder cases. TechCrunch reports https://techcrunch.com/2026/10/22/microsofts-mustafa-suleyman-says-its-ai-outperforms-doctors-on-hard-cases-as-he-chases-medical-superintelligence/ that it lifted Gemini 2.5 Flash from 46.4% to nearly 69%, and in Microsoft’s later tests took the newer GPT-5.5 from 72% to nearly 87% on the NEJM cases. On the emergency cases GPT-5.5 scored about 85% with or without it. Why the numbers changed since 2025 Microsoft first posted the work as a preprint in June 2025 https://arxiv.org/abs/2506.22405 , when MAI-DxO ran on OpenAI’s o3 and scored up to 85.5% against 20% for 21 doctors from the US and UK. Those doctors weren’t allowed to look anything up. The published version runs on GPT-5, adds a group of doctors allowed textbooks and the internet, and adds the emergency department cases, which is why the comparison Microsoft now leads with is 83% against 36%. ‘It needs clinical validation’ MAI-DxO has only been tested on past cases, and Microsoft says the real test comes next. With Beth Israel Deaconess, it plans a study in which consenting patients get normal care and also talk to the system, whose recommendations will be compared with their doctors’ independent assessments. Dominic King, Microsoft AI’s vice president of health and a co-author of the paper, told The Verge it will likely “get underway in the next couple of months”. Suleyman told The Verge https://www.theverge.com/ai-artificial-intelligence/1016690/suleyman-microsoft-diagnostic-orchestrator-science-medical-superintelligence : It needs clinical validation. That is a leap, but this is a major milestone along that journey. Mustafa Suleyman, CEO, Microsoft AI, to The Verge Suleyman also told TechCrunch that Microsoft is working with Mayo Clinic on a healthcare foundation model planned for next year, part of a push that started with Microsoft training its own frontier models https://madrobot.blog/2026/10/16/microsoft-mai-base-2-2-8-trillion-parameter-ai-model-deepseek/ . The company is open-sourcing SDBench for a selection of the NEJM cases so other researchers can test their own systems and check its results. Governments are moving too: Britain this month agreed to bring AI into NHS hospitals sooner https://madrobot.blog/2026/10/07/uk-accepts-all-44-nhs-ai-commission-recommendations-mhra-general-purpose-models/ . Why it matters A peer-reviewed paper in Science gives Microsoft’s claim more weight than a company benchmark, and it already reads like a pitch for clinical decision tools. But the doctors it beat were generalists working alone on cases built to be hard, and the hospital study is what will show whether any of this helps real patients beyond one-off stories of chatbots spotting rare conditions https://madrobot.blog/2026/09/25/chatgpt-spotted-toddler-rare-condition-msmds-doctors-never-heard-of/ . Sources: Nori et al., “Sequential diagnosis with language models”, Science October 22, 2026 https://www.science.org/doi/10.1126/science.aeh7930 ; Microsoft AI, “Building Medical Superintelligence” https://microsoft.ai/news/building-medical-superintelligence/ ; arXiv preprint June 2025 https://arxiv.org/abs/2506.22405 ; The Verge https://www.theverge.com/ai-artificial-intelligence/1016690/suleyman-microsoft-diagnostic-orchestrator-science-medical-superintelligence and TechCrunch https://techcrunch.com/2026/10/22/microsofts-mustafa-suleyman-says-its-ai-outperforms-doctors-on-hard-cases-as-he-chases-medical-superintelligence/ interviews.