The Gaps in Clinical Reasoning of AI: A Closer Look at Limitations

A new evaluation of AI language models in hematologic oncology found that the best models achieved only 68% accuracy, with data utilization dropping from 57% to 26% across three rounds. The models exhibited cognitive biases similar to novice clinicians, such as search satisficing and anchoring, highlighting a gap between reasoning ability and clinical decision-making under uncertainty.

The Gaps in Clinical Reasoning of AI: A Closer Look at Limitations AI models in oncology show promise but fail in decisive clinical reasoning. While they boast 68% accuracy, they stumble in effective data gathering. AI models have come a long way in medicine, but they're hitting a wall in practical application within clinical settings. Despite scoring well on medical knowledge exams, these models struggle with something inherently human: making informed decisions under uncertainty. AI's Struggle with Decision-Making In a recent evaluation /glossary/evaluation focused on hematologic oncology, a field riddled with complexity, AI language models were put to the test. They had to request necessary clinical data across three rounds before settling on a diagnosis and treatment plan. Here's what the benchmarks actually show: the best models reached only 68% overall accuracy. Notably, information utilization was a important factor. The models started strong, utilizing 57% of the available data. But by the final round, utilization plummeted to a mere 26%. This oversight left molecular and cytogenetic data, turning point for treatment decisions, unexamined. It's clear, the architecture matters more than the parameter /glossary/parameter count. This stark decline in data usage paints a picture of models that haven't yet mastered strategic information-seeking. What Lies Beneath the Surface While reasoning /glossary/reasoning traces of these AI systems scored impressively on clinical reasoning rubrics with 91% exceeding the threshold , these scores didn't correlate with actual accuracy. This discrepancy reveals a gap. They can string together coherent rationales, but can't always arrive at the correct conclusions. Simply put, they get lost in translation between logic and real-life application. So, why should this matter? Because the reality is, AI is mirroring the same cognitive biases seen in novice clinicians. Errors like search satisficing, anchoring, and premature closure emerged as dominant failure modes. If AI models are to become reliable partners in healthcare, they need more than just a database of knowledge. They need to think more like seasoned clinicians. The Road Ahead for AI in Healthcare The present limitation of AI in clinical oncology isn't due to a lack of medical knowledge. It's the failure to adequately seek out information under uncertain conditions. These findings should serve as a wake-up call for developers and researchers. The goal must shift toward improving how these models gather and assess data in uncertain environments. Can AI truly replace the nuanced decision-making of human clinicians? Not yet. However, with strategic improvements and a deeper understanding of decision-making processes, AI could one day serve as a reliable assistant in healthcare. Get AI news in your inbox Daily digest of what matters in AI. Key Terms Explained Evaluation /glossary/evaluation The process of measuring how well an AI model performs on its intended task. Parameter /glossary/parameter A value the model learns during training — specifically, the weights and biases in neural network layers. Reasoning /glossary/reasoning The ability of AI models to draw conclusions, solve problems logically, and work through multi-step challenges.