{"slug": "i-m-93-confident-is-a-string-not-a-probability", "title": "\"I'm 93% confident\" is a string, not a probability", "summary": "A developer argues that LLM-reported confidence scores are generated text rather than calibrated probabilities, and that treating them as probabilities turns false precision into automated risk. The post shows how to measure calibration with coverage-accuracy curves and per-bin calibration checks, and warns that a single global threshold can mask failures in specific slices such as language or OCR quality. It recommends validating any confidence signal—verbal, log-prob, ensemble agreement, or verifier—as a hypothesis and setting per-slice thresholds where curves diverge.", "body_md": "In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93. You write `if confidence > 0.9: auto_approve()` and move on.\n\nIn production, that number is just more generated text. Nobody measured it against outcomes. It looks like a probability, it's formatted like a probability, and your code treats it like a probability. That is how false precision becomes automated risk.\n\nA confidence score earns the word \"confidence\" only when it is calibrated: of all the cases where the system said ~90%, roughly 90% were actually correct. Until you've checked that on your own data, the number is decoration.\n\nIllustrative scenario: a team routes invoice classification through an LLM and auto-processes anything above 0.9. On their eval set, accuracy is 94%. Looks great.\n\nWhat they never measured is accuracy *within* the >0.9 bucket. When someone finally slices it, the high-confidence bucket is barely better than the low-confidence one. The model emits 0.9+ on almost everything. The threshold wasn't filtering anything; it was a rubber stamp with a decimal point.\n\nIf your system can defer (hand off to a human, ask for clarification, fall back to a rule), then you're not evaluating one number. You're evaluating a trade-off:\n\nPush the threshold up and coverage drops while selective accuracy (hopefully) rises. Whether it actually rises is the whole question. Here's the minimal version of measuring it:\n\n``` python\nimport numpy as np\n\ndef coverage_accuracy_curve(conf, correct, thresholds):\n    conf, correct = np.asarray(conf), np.asarray(correct)\n    rows = []\n    for t in thresholds:\n        answered = conf >= t\n        coverage = answered.mean()\n        sel_acc = correct[answered].mean() if answered.any() else float(\"nan\")\n        rows.append((t, round(coverage, 3), round(sel_acc, 3)))\n    return rows\n\ndef calibration_bins(conf, correct, n_bins=10):\n    conf, correct = np.asarray(conf), np.asarray(correct)\n    edges = np.linspace(0, 1, n_bins + 1)\n    for lo, hi in zip(edges[:-1], edges[1:]):\n        m = (conf >= lo) & (conf < hi)\n        if m.any():\n            print(f\"[{lo:.1f}, {hi:.1f})  n={m.sum():4d}  \"\n                  f\"stated={conf[m].mean():.2f}  actual={correct[m].mean():.2f}\")\n```\n\nIf \"stated\" and \"actual\" drift apart in the bins you care about, your threshold is lying to you.\n\nEven a calibrated score can be calibrated *on average* and miscalibrated everywhere that matters. Illustrative scenario: overall, the 0.9 bucket is 91% correct. Split by language and the English tickets sit at 95% while the Greek and German ones sit far lower. Split by input quality and the clean PDFs are fine while the scanned, OCR'd ones fall apart.\n\nOne threshold across all of that averages away exactly the failures you built the deferral path to catch. Run the same curve per slice:\n\n```\nfor slice_name, idx in slices.items():   # e.g. by task type, language, source quality\n    print(slice_name, coverage_accuracy_curve(conf[idx], correct[idx], [0.7, 0.8, 0.9]))\n```\n\nIf the curves differ meaningfully, you need per-slice thresholds, or you need to stop trusting the score for that slice at all.\n\nAsking the model \"how sure are you?\" and reading token log-probs are different signals, and neither is calibrated by default. Self-reported numbers in particular tend to cluster on round, reassuring values. Treat any source of confidence (verbal, log-prob, ensemble agreement, a separate verifier) as a hypothesis to test, not a fact to consume.\n\nConfidence isn't a model property. It's a system property you have to measure. The real question isn't \"does the model feel sure?\" It's \"is the system's decision to proceed reliable for *this* task, in *this* context, on *this* kind of input?\"\n\nHow are you deciding when your LLM pipeline is allowed to act without a human? Is anyone here actually running per-slice thresholds in production, or is everyone on a single global cutoff?", "url": "https://wpnews.pro/news/i-m-93-confident-is-a-string-not-a-probability", "canonical_source": "https://dev.to/cyclopt_dimitrisk/im-93-confident-is-a-string-not-a-probability-2ndc", "published_at": "2026-10-22 06:30:29+00:00", "updated_at": "2026-10-22 06:40:50.305258+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-agents", "mlops", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-m-93-confident-is-a-string-not-a-probability", "markdown": "https://wpnews.pro/news/i-m-93-confident-is-a-string-not-a-probability.md", "text": "https://wpnews.pro/news/i-m-93-confident-is-a-string-not-a-probability.txt", "jsonld": "https://wpnews.pro/news/i-m-93-confident-is-a-string-not-a-probability.jsonld"}}