Microsoft says its medical AI system can diagnose difficult cases more than twice as accurately as practicing doctors.
Dubbed the Microsoft AI Diagnostic Orchestrator (MAI-DxO), the system “substantially outperformed practicing generalist physicians” in diagnostic accuracy while ordering fewer tests and procedures, according to a newly published paper.
The Microsoft study, titled “Sequential Diagnosis with Language Models,” was first posted online last year on arXiv as a non-peer-reviewed preprint. On Thursday, the findings were finally published in the peer-reviewed journal Science.
That’s a major step toward validating the findings, but any real-world deployment would still require clinical testing, Microsoft’s AI chief Mustafa Suleyman told The Verge. Microsoft already has plans for that: the company’s AI health lead, Dr. Dominic King, told the publication that testing will begin at Beth Israel Deaconess Medical Center in Boston, Massachusetts, within the next couple of months.
In tests involving complex medical cases published in The New England Journal of Medicine, DxO achieved 83.2% diagnostic accuracy, compared with 36.1% for generalist physicians who had access to reference materials. When physicians were denied those resources, their accuracy dropped to roughly 22%, making DxO nearly four times as accurate. In a separate test involving emergency department admissions, Microsoft’s AI again outperformed generalist physicians, though by a narrower margin of 80.3% to 46.8%. The system achieved its strongest results using OpenAI’s GPT-5, with additional testing showing improvements using GPT-5.5.
“It also manages to do this at half or even one third of the cost, getting to the right diagnosis with fewer tests and procedures,” Microsoft said in a blog post Thursday. Those savings, however, are based on estimated costs in simulated cases rather than actual medical expenses. The researchers admitted that they “would expect the performance gap to narrow” if DxO were compared with expert specialists rather than generalists.
Notably, DxO works sequentially, meaning patients don’t simply present the system with a comprehensive list of symptoms, medical history, and test results. Instead, it starts with a chief complaint and must determine which follow-up questions to ask and tests to order before arriving at a diagnosis. That’s much closer to how doctors actually diagnose patients. The researchers say benchmarks have previously failed to evaluate LLMs’ diagnostic capabilities sequentially and propose a new benchmark called the Sequential Diagnosis Benchmark, or SDBench.
DxO “simulates a panel of physicians,” the paper explains, “continually revising the differential diagnosis and selecting cost-effective tests.” The system assigns different roles to AI agents that propose diagnoses, select tests, challenge assumptions, and weigh costs. Researchers suggest this approach can help counter some of the cognitive biases that affect individual physicians, such as anchoring on an initial diagnosis or overlooking alternative explanations.
Similar claims have been made about other companies’ AI tools. Earlier this year, researchers from Harvard University and Beth Israel Deaconess Medical Center found that OpenAI’s o1-preview reasoning model outperformed physicians in diagnosing emergency room patients during triage.
In all these cases, medical experts emphasize that superior performance on diagnostic tests doesn’t mean AI systems can or should replace human doctors. Instead, the tools are being positioned as assistants to medical professionals. Even if DxO performs well in clinical trials, widespread adoption would raise difficult legal questions about accountability when an AI-assisted diagnosis goes wrong. And for what it’s worth, a 2025 study found that doctors who relied on AI to identify potential cancer risks during colonoscopies performed worse than before once the technology was taken away.