{"slug": "are-you-sure-your-llm-is-sure", "title": "Are You Sure Your LLM Is Sure?", "summary": "A paper submitted to arXiv on 15 October 2026 by Ramya Hebbalaguppe and co-authors introduces RADAR (Repair, Adopt, and Diagnose After Recalibration), a framework that separates repairing LLM confidence scores from deciding whether to deploy the repair and auditing the errors that remain. The paper proposes Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts, and reports that across 47,400 replayed raw-score allocations, paired betting adopted candidates with lower held-out test risk in 42.95% of allocations versus 15.09% for one-sided empirical Bernstein. The authors argue that richer predictors can improve confidence estimation but projecting their information back to a scalar score can lose those gains, and that lower prediction risk need not yield better review decisions.", "body_md": "# Computer Science > Artificial Intelligence\n\n  [Submitted on 15 Oct 2026]\n\n# Title:Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence\n\n[View PDF](https://arxiv.org/pdf/2610.22139)\n\n[HTML (experimental)](https://arxiv.org/html/2610.22139v1)\n\nAbstract:Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer predictors can improve confidence estimation, but projecting their information back to a scalar score can lose these gains, and lower prediction risk need not yield better review decisions. Across 47,400 replayed raw-score allocations, paired betting adopts candidates with lower held-out test risk in 42.95% of allocations, versus 15.09% for one-sided empirical Bernstein. RADAR thus provides a principled workflow to repair what a scalar score can express, adopt a repair only when supported by evidence, and diagnose systematic errors that recalibration leaves behind. Code and data will be released upon acceptance.\n    \n\n## Submission history\n\nFrom: Ramya Hebbalaguppe [\n[view email](https://arxiv.org/show-email/ac726b30/2610.22139)]\n\n**[v1]** Thu, 15 Oct 2026 16:12:29 UTC (198 KB)\n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/are-you-sure-your-llm-is-sure", "canonical_source": "https://arxiv.org/abs/2610.22139", "published_at": "2026-10-17 13:39:04+00:00", "updated_at": "2026-10-17 14:03:11.454571+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety", "machine-learning"], "entities": ["RADAR", "Probe-Anchored Isotonic Calibration", "Ramya Hebbalaguppe", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/are-you-sure-your-llm-is-sure", "markdown": "https://wpnews.pro/news/are-you-sure-your-llm-is-sure.md", "text": "https://wpnews.pro/news/are-you-sure-your-llm-is-sure.txt", "jsonld": "https://wpnews.pro/news/are-you-sure-your-llm-is-sure.jsonld"}}