"I'm 93% confident" is a string, not a probability

A developer argues that LLM-reported confidence scores are generated text rather than calibrated probabilities, and that treating them as probabilities turns false precision into automated risk. The post shows how to measure calibration with coverage-accuracy curves and per-bin calibration checks, and warns that a single global threshold can mask failures in specific slices such as language or OCR quality. It recommends validating any confidence signal—verbal, log-prob, ensemble agreement, or verifier—as a hypothesis and setting per-slice thresholds where curves diverge.

In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93. You write if confidence 0.9: auto approve and move on. In production, that number is just more generated text. Nobody measured it against outcomes. It looks like a probability, it's formatted like a probability, and your code treats it like a probability. That is how false precision becomes automated risk. A confidence score earns the word "confidence" only when it is calibrated: of all the cases where the system said ~90%, roughly 90% were actually correct. Until you've checked that on your own data, the number is decoration. Illustrative scenario: a team routes invoice classification through an LLM and auto-processes anything above 0.9. On their eval set, accuracy is 94%. Looks great. What they never measured is accuracy within the 0.9 bucket. When someone finally slices it, the high-confidence bucket is barely better than the low-confidence one. The model emits 0.9+ on almost everything. The threshold wasn't filtering anything; it was a rubber stamp with a decimal point. If your system can defer hand off to a human, ask for clarification, fall back to a rule , then you're not evaluating one number. You're evaluating a trade-off: Push the threshold up and coverage drops while selective accuracy hopefully rises. Whether it actually rises is the whole question. Here's the minimal version of measuring it: python import numpy as np def coverage accuracy curve conf, correct, thresholds : conf, correct = np.asarray conf , np.asarray correct rows = for t in thresholds: answered = conf = t coverage = answered.mean sel acc = correct answered .mean if answered.any else float "nan" rows.append t, round coverage, 3 , round sel acc, 3 return rows def calibration bins conf, correct, n bins=10 : conf, correct = np.asarray conf , np.asarray correct edges = np.linspace 0, 1, n bins + 1 for lo, hi in zip edges :-1 , edges 1: : m = conf = lo & conf < hi if m.any : print f" {lo:.1f}, {hi:.1f} n={m.sum :4d} " f"stated={conf m .mean :.2f} actual={correct m .mean :.2f}" If "stated" and "actual" drift apart in the bins you care about, your threshold is lying to you. Even a calibrated score can be calibrated on average and miscalibrated everywhere that matters. Illustrative scenario: overall, the 0.9 bucket is 91% correct. Split by language and the English tickets sit at 95% while the Greek and German ones sit far lower. Split by input quality and the clean PDFs are fine while the scanned, OCR'd ones fall apart. One threshold across all of that averages away exactly the failures you built the deferral path to catch. Run the same curve per slice: for slice name, idx in slices.items : e.g. by task type, language, source quality print slice name, coverage accuracy curve conf idx , correct idx , 0.7, 0.8, 0.9 If the curves differ meaningfully, you need per-slice thresholds, or you need to stop trusting the score for that slice at all. Asking the model "how sure are you?" and reading token log-probs are different signals, and neither is calibrated by default. Self-reported numbers in particular tend to cluster on round, reassuring values. Treat any source of confidence verbal, log-prob, ensemble agreement, a separate verifier as a hypothesis to test, not a fact to consume. Confidence isn't a model property. It's a system property you have to measure. The real question isn't "does the model feel sure?" It's "is the system's decision to proceed reliable for this task, in this context, on this kind of input?" How are you deciding when your LLM pipeline is allowed to act without a human? Is anyone here actually running per-slice thresholds in production, or is everyone on a single global cutoff?