Microsoft’s new AI is surpassing the abilities of human doctors — according to its own study, at least.
The research, which was peer-reviewed and published in the prestigious journal Science, showed that the medical model, dubbed the Microsoft AI Diagnostic Orchestrator (MAI-DxO), was over twice as accurate than generalist physicians at diagnosing patients. And unlike previous studies, this one forced the AI to approach each case like an actual doctor would, instead of being armed with all the facts right away.
The tests were conducted in a controlled benchmark setting, meaning its usefulness in a true medical environment is up for debate. Still, Microsoft was eager to declare victory.
The company’s CEO of its AI lab Mustafa Suleyman said the age of AI improving lives through healthcare “is finally here,” telling The Verge that further advances are right around the corner.
“Next year we’re going to see extremely impressive deployments of intelligent systems in production,” Suleyman said in an interview with the outlet. “I think that this is just going to change the way care is delivered in a big way.”
For the study, the researchers had the DxO model examine 304 clinicopathological conference cases from the New England Journal of Medicine and 325 emergency department-to-admission encounters from a database called MIMIC-IV-ED — in other words, a near fifty-fifty mix of case reports and emergency department admissions. With the former cohort, DxO achieved an 83.2 percent diagnostic accuracy, compared to 36.1 percent for generalist physicians who examined the same cases and were allowed to view reference materials. With the emergency department cases, the AI scored an 80.3 percent accuracy, while the human doctors came in at 46.8 percent.
The AI models weren’t just graded by the accuracy of their diagnoses, but the estimated costs of the tests ordered to reach them. And DxO, overall, managed to do this at half or even one third of the cost of the human doctors, Microsoft said in a blog post.
The findings, if borne out, are an impressive testament to AI’s capabilities and usefulness in medical applications. Still, there’s a lot more to being a doctor than analyzing symptoms. Some of the hardest work is getting patients to open up about what they’re experiencing, and pry for stuff they may not have even thought worth mentioning,
The study addresses this, too — or at least somewhat — by having DxO work sequentially. Rather than being given a list of complete symptoms, the AI has to work off a patient’s initial complaint, come up with a working diagnosis, and choose follow up questions to ask and decide what tests to order while considering the costs involved.
Previous studies “have not tested diagnosis as an active, sequential information-gathering process that jointly considers diagnostic accuracy and diagnostic workup costs,” the authors wrote, and instead fed the complete case in a single prompt.
But the study could only imperfectly simulate the back and forth a doctor has with a patient by having DxO talk to a “gatekeeper agent” that “mediates all information requests drawing from the underlying case record.” Put another way, it’s an AI diagnosing a patient that’s played by an AI. And while there’s no doubt that AI models are powerful analyzers and synthesizers of information, human doctors already did the leg work of putting together these neat, highly detailed case files for the AI to pore over in the first place.
More on medical AI: Startup Deploys AI-Engineered Virus on Human Patient