The Atlantic reports that a team of primarily Harvard and Stanford researchers announced the results of a study in April that was linked to publication in the journal Science, that compared ChatGPT against hundreds of physicians in a diagnostic obstacle course using written medical cases and real-world patient information, and that the model outperformed the clinicians. The Atlantic quotes lead author Adam Rodman saying, "I get a little bit queasy about how some of these results might be used," at a press conference ahead of the paper's publication. The Atlantic's author, a practicing pathologist, reports receiving an email from their medical-center administrators notifying clinicians that an "AI-powered clinical reasoning tool" is now available, and the article notes that at least one generative-AI product has received FDA approval.
What happened
The Atlantic reports that a team of primarily Harvard and Stanford researchers announced the results of a study in April that was linked to publication in the journal Science that tested ChatGPT against hundreds of physicians in a diagnostic obstacle course using written medical mysteries and information from real-world patients, and that the model outperformed the clinicians. The Atlantic quotes lead author Adam Rodman saying, "I get a little bit queasy about how some of these results might be used," at a press conference ahead of the paper's publication. The Atlantic's author, a practicing pathologist, reports receiving an email from medical-center administrators announcing that an "AI-powered clinical reasoning tool" was now available to clinicians. The Atlantic also reports that multiple generative-AI products have been rolled out in health systems and that at least one tool has received FDA approval.
Technical details
The Atlantic describes the study as a diagnostic obstacle course built from written clinical cases and real-world patient information; the article does not provide architecture-level disclosures about the model beyond naming ChatGPT. The paper was presented and discussed publicly at a press conference linked to its Science publication, per The Atlantic.
Editorial analysis
Industry context: When controlled evaluations show large-model assistants outperforming clinicians on written diagnostic tasks, public and institutional interest in clinical decision-support accelerates. Observed patterns in similar transitions indicate rapid procurement and pilot deployments often outpace robust prospective validation, governance, and integration work. For practitioners, that gap typically manifests as increased demands on data pipelines, EHR integration effort, and new monitoring/alerting requirements.
What to watch
Indicators an observer should follow include:
- •uptake by electronic-health-record vendors and major hospital systems;
- •number and scope of FDA clearances or approvals for clinical-use generative-AI tools; - •results from prospective, randomized clinical trials testing patient-centered outcomes; and
- •reports of real-world safety incidents or clinician-reported harms or workflow disruptions.
Scoring Rationale #
A controlled study reported in Science showing a large-model assistant outperforming clinicians, combined with active rollouts in hospitals and FDA activity, is notable for practitioners. It raises integration, validation, and safety monitoring issues that affect data pipelines and clinical workflows.
Practice with real Health & Insurance data
90 SQL & Python problems · 15 industry datasets
250 free problems · No credit card
See all Health & Insurance problems