The Safe-Psych benchmark evaluates large language models in handling diagnostic uncertainty in psychiatry, revealing their limitations. The results show a tendency to misdiagnose or overly abstain.
Large language models (LLMs) are quickly infiltrating the healthcare space, but a recent study throws a wrench in their perceived reliability, especially in psychiatry. The study introduces Safe-Psych, a benchmark designed to test how these models manage evolving diagnostic uncertainty. In clinical terms, it's about how well models react when the full picture isn't immediately clear.
The Safe-Psych Benchmark #
Safe-Psych isn't just a catchy name. It's a comprehensive tool, comprising over 1,000 real-world psychiatric clinical notes. These notes are broken down to simulate the gradual disclosure of information, mimicking real-life scenarios where a complete diagnosis can't be made at the get-go. Psychiatrist-derived action labels guide the evaluation: DIAGNOSE, CLARIFY, or ABSTAIN.
The regulatory detail everyone missed: The benchmark forces LLMs to decide their next move at each stage. Should they diagnose despite incomplete data, ask for more information, or simply abstain from making a judgment?
Model Performance: Not Up to Par #
Even state-of-the-art models show glaring shortcomings. The results are clear. Over 60% of the time, these models under-abstain, meaning they make diagnoses without sufficient evidence. And when models are primed to be safety-aware, they err on the side of caution, abstaining more than necessary. This misplaced caution only shifts the errors from one form to another.
Surgeons I've spoken with say similar issues arise in robotic-assisted surgeries, where hesitation can be just as problematic as premature action.
Why It Matters #
Why should you care? Because the potential for AI to transform psychiatry is immense, but these findings suggest we're not there yet. If AI can't handle diagnostic uncertainty effectively, it can't be trusted with real-world applications where patient outcomes are at stake.
Here's the kicker: During sequential evaluation, models often jump the gun, diagnosing before gathering enough evidence. They're not asking the right questions unless specifically prompted to do so, leading to less accurate outcomes than when they wait for complete data.
The Path Forward #
Safe-Psych is now available to the research community, aiming to spur improvements in LLM safety within healthcare. The FDA pathway matters more than the press release. For companies working on AI in healthcare, this benchmark should serve as a wake-up call. The next steps involve refining these models to better handle uncertainty without swinging between rash decisions and excessive caution.
In the end, the potential for AI to revolutionize psychiatry is real, but only if it moves beyond its current limitations. It's about time these models learn the importance of saying 'I don't know yet.'
Get AI news in your inbox
Daily digest of what matters in AI.