LLMs Fumble at Explaining Probabilities: Consistency Without Calibration

Researchers evaluating nine large language models in a two-stage prediction pipeline found that while LLMs produce consistent verbal descriptions of probabilistic outputs, they are miscalibrated and fail to accurately reflect underlying probabilities, especially for uncertainty. The study highlights that current LLMs are not ready for standalone risk communication in high-stakes domains like healthcare and finance.

LLMs Fumble at Explaining Probabilities: Consistency Without Calibration LLMs are tasked with explaining AI predictions, but their knack for consistency falls flat calibration. They're good, but not quite good enough. Large Language Models LLMs are trying to wear many hats, including explaining AI-generated outputs. But communicating probabilistic information in plain language, they're still stumbling. If these models are going to be our go-to explainers, they need to nail two things: consistency and accuracy. Identical inputs should yield identical responses, and those responses should mirror the numbers behind them. Simple, right? Testing the Waters To test this, researchers evaluated nine LLMs in a two-stage prediction pipeline. First, an upstream model spat out probabilistic outputs characterized by likelihood and uncertainty. Then, LLMs had to pick the right words to match these probabilities. The experiment simulated predictions using a Beta distribution, shaking things up with six different domain contexts and ten temperature /glossary/temperature settings. Each scenario? Repeated ten times. The results show LLMs are consistent but miscalibrated. They handle likelihood tasks better than uncertainty ones. Even when given precomputed summary stats like mode and prior sample size, the models weren't immune to the context. The sticking point? Verbalization. It's not just about crunching numbers. it's about explaining them right. Where's the Problem? Let's be blunt. Current LLMs aren't ready to be standalone tools for risk communication. Sure, they can spit out consistent descriptions, but if those descriptions aren't accurate reflections of the underlying probabilities, what's the point? The models' inability to handle uncertainty effectively makes them unreliable for zero-shot predictions. The speed difference isn't theoretical. You feel it when these models attempt to explain the unexplainable and fall short. They need more than just data to get it right. They need to understand the nuance of human language and context better. If they're to replace human judgment, they've got a long road ahead. Why Should We Care? Why does this matter? Because as more AI models enter domains where risk communication is key, think healthcare or finance, the need for reliable, transparent explanations grows. If LLMs can't bridge this gap, we risk losing trust in AI systems. Are we ready to hand over critical decision-making to models that might misinterpret or miscommunicate probabilities? Open weights don't wait for permission, and neither should we holding these models accountable. The tech is advancing, but if you haven't run it locally yet, you're late. We need better calibration, better verbalization, and a better understanding of where these systems fall short before they can be trusted in high-stakes situations. Get AI news in your inbox Daily digest of what matters in AI.