AI gets more diagnoses right than generalist doctors in medical challenge

Microsoft researchers reported in Science that their Diagnostic Orchestrator (DxO) tool, which simulates a panel of collaborating clinical experts, diagnosed 83.2% of 304 published New England Journal of Medicine cases correctly and 80.3% of 325 emergency department admission records, versus 36.1% and 46.8% for individual generalist physicians tested through the SDBench framework. The authors cautioned that "these comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice," noting SDBench measures diagnostic reasoning under idealized information conditions rather than bedside performance or patient outcomes. The team said future work should explore doctor-AI collaboration and use of the technology in medical education and training.

AI gets more diagnoses right than generalist doctors in medical challenge Paul Arnold Author Lisa Lock Scientific Editor Robert Egan Senior Editor One of the many challenging parts of being a doctor is diagnosing exactly what is wrong with a patient. This is because numerous illnesses often have overlapping symptoms, which can make the diagnostic journey a long one. In addition to asking patients questions, doctors may need to order tests and consult medical references or even other professionals. Obviously, for the individual patient, the sooner a diagnosis is reached, the sooner they can start receiving the most appropriate treatment. Now, in yet another breakthrough for AI, Microsoft researchers claim they have developed a tool that can outperform individual generalist physicians in diagnosing complex medical cases. It is called DxO, which stands for Diagnostic Orchestrator, and simulates a panel of clinical experts who collaborate to reach a diagnosis. The diagnostic challenge To see how well DxO performs and how it compares with medical professionals, researchers developed a testing framework called SDBench, which was essentially an interactive quiz. The team took 304 published diagnostic cases from the New England Journal of Medicine and 325 emergency department admission records. They turned each of these patient files into a step-by-step test. At the start of each test, a doctor or DxO was given a summary called a clinical vignette. This included basic details like the patient's age and sex and their main symptoms. For example, one case started with: "A 29-year-old woman was admitted to the hospital because of sore throat and peritonsillar swelling and bleeding. Symptoms did not abate with antimicrobial therapy." An AI system acted as a gatekeeper, providing information from the original case file in response to questions and test requests. Once a doctor or DxO felt they had enough information, they submitted a final diagnosis. An AI judge graded the diagnosis against the one recorded in the original case file. The results are in The difference between AI and real doctors was striking, as the team reports in a paper published https://www.science.org/doi/10.1126/science.aeh7930 in the journal Science . Regarding the complex published cases, DxO made an accurate diagnosis 83.2% of the time and achieved 80.3% accuracy on emergency department admission cases. Human doctors, on the other hand, averaged 36.1% and 46.8%, respectively. There were also clear differences in approach. DxO asked more questions about the patient's background upfront before ordering tests, which could help reduce testing costs. While AI appears to have come out on top, the researchers state that it doesn't mean it is better than doctors in real-world practice. "These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice." Additionally, the Microsoft team did not conduct the testing in everyday health care settings. "As a result, SDBench measures diagnostic reasoning under idealized information conditions rather than bedside performance or patient outcomes." The study authors say future work should explore how doctors and AI systems could work together. They also suggest the technology could be used in medical education and training. Written for you by our author Paul Arnold https://sciencex.com/help/editorial-team/paul-arnold/ , edited by Lisa Lock https://sciencex.com/help/editorial-team/lisa-lock/ , and fact-checked and reviewed by Robert Egan https://sciencex.com/help/editorial-team/robert-egan/ —this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive. If this reporting matters to you, please consider a donation https://sciencex.com/donate/?utm source=story&utm medium=story&utm campaign=story especially monthly . You'll get an ad-free account as a thank-you. Publication details Harsha Nori et al., Sequential diagnosis with language models, Science 2026 . DOI: 10.1126/science.aeh7930 https://dx.doi.org/10.1126/science.aeh7930 Journal information: New England Journal of Medicine https://medicalxpress.com/journals/new-england-journal-of-medicine/ , Science https://medicalxpress.com/journals/science/ Key medical concepts Diagnostic Accuracy Clinical categories Family medicine © 2026 Science X Network Citation : AI gets more diagnoses right than generalist doctors in medical challenge 2026, October 23 retrieved 24 October 2026 from https://medicalxpress.com/news/2026-10-ai-generalist-doctors-medical.html