Claude Fable 5, Anthropic's latest model, shines in accuracy but often refuses to engage with certain biomedical questions. What's driving this cautious behavior?
AI, models often get judged by how they handle complex tasks. Anthropic's Claude Fable 5 is no exception. This model, Anthropic's most capable publicly available iteration, has been put to the test across eight biomedical benchmarks. Here's the kicker: while Fable 5 exhibits remarkable accuracy, it also shows an intriguing tendency to refuse answering a significant portion of questions.
The Refusal Phenomenon #
Think of it this way: Claude Fable 5 is like a brilliant student who aces everything but skips questions they find distasteful or tricky. According to recent evaluations, the model refused to answer between 8.0% and a staggering 99.4% of questions, depending on the benchmark. That's unlike its predecessors and GPT-5, which didn't show similar patterns.
Why is this important? If you've ever trained a model, you know the balance between accuracy and engagement is essential. For Fable 5, the main issue isn't its capability, it's willingness. Once you filter out the refused queries, its accuracy is top-tier, beating out other models in this study. But what drives these refusals?
Unpacking the Patterns #
Upon closer inspection, two distinct refusal patterns emerge. First, there's a trend in basic-science and mechanism-heavy content, especially notable in benchmarks like MedQA and MedXpertQA MM. This pattern was consistent across other evaluations using each benchmark's category labels.
Then there's the disease-domain pattern. Particularly on RareBench, Fable 5 tends to refuse questions about inborn metabolic diseases almost universally, while it readily engages with adult-onset autoimmune scenarios. It's like having a model that excels in calculus but skips algebra.
What Does This Mean for AI in Medicine? #
Here's why this matters for everyone, not just researchers: the practical application of AI in fields like biomedicine hinges on models both understanding and engaging with the material. While Claude Fable 5's accuracy is commendable, the refusal to engage is a limitation. The analogy I keep coming back to is a doctor who's brilliant but occasionally refuses to treat certain conditions.
If AI models are to assist in real-world medical diagnoses or research, they need to be both willing and able to address all relevant queries. Otherwise, their utility is severely constrained. So here's the thing: will future iterations of Claude or other models overcome this reluctance? Will they evolve to tackle all challenges head-on? Only time, and more training, will tell. But, as it stands, Fable 5 is a genius with a curious blind spot.
Get AI news in your inbox
Daily digest of what matters in AI.
Key Terms Explained #
Anthropic An AI safety company founded in 2021 by former OpenAI researchers, including Dario and Daniela Amodei.
Benchmark A standardized test used to measure and compare AI model performance.
Claude Anthropic's family of AI assistants, including Claude Haiku, Sonnet, and Opus.
GPT Generative Pre-trained Transformer.