{"slug": "free-lunch-for-llms-confidence-the-rl-critic-already-knows", "title": "Free Lunch for LLMs Confidence: The RL Critic Already Knows", "summary": "A new arXiv paper (2610.31820v1) proves that confidence estimation and critic training in reinforcement learning are mathematically equivalent under a correctness-based outcome, meaning LLMs trained with value-based RL can reuse their existing critic as a confidence estimator with no additional training or data. Across math, code, tool calling, and general reasoning spanning a dozen domains, the reused critic was the best confidence estimator on every in-distribution split and the best-calibrated on 14 of 16, while the policy remained unchanged and task performance was preserved. The authors also show an external model trained solely on a policy's behavioral traces can match the policy's own confidence accuracy without access to its internal representations.", "body_md": "arXiv:2610.31820v1 Announce Type: new \nAbstract: Reliable confidence estimation is essential for trustworthy deployment of large language models (LLMs), but dedicated confidence training stage requires additional data and compute. We ask a fundamental question: what prediction problem does confidence training actually solve? Inspired by prior work on external confidence estimation, we compare self- and external estimators of comparable capacity trained on the same (question, response, correctness) triples. We show that an external model trained solely on a policy's behavioral traces can estimate the policy's confidence as accurately as the policy itself, without accessing the policy's internal representations. This points to a deeper connection with value function learning in reinforcement learning (RL): under a correctness-based outcome, we prove that confidence estimation and critic training are equivalent, both predicting the likelihood that a policy's response will be correct from its trajectory. A simple consequence follows: for LLMs trained with value-based RL, the existing critic can be reused as an off-the-shelf confidence estimator with no additional training or data. The policy remains unchanged, so its task performance is preserved, and the critic provides confidence throughout generation, including intermediate reasoning steps. Across math, code, tool calling, and general reasoning spanning a dozen domains, the reused critic is the best confidence estimator on every in-distribution split and the best-calibrated on 14 of 16, without hurting the task performance.", "url": "https://wpnews.pro/news/free-lunch-for-llms-confidence-the-rl-critic-already-knows", "canonical_source": "https://arxiv.org/abs/2610.31820", "published_at": "2026-10-26 04:00:00+00:00", "updated_at": "2026-10-26 04:13:08.986597+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-safety"], "entities": ["arXiv", "2610.31820v1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/free-lunch-for-llms-confidence-the-rl-critic-already-knows", "markdown": "https://wpnews.pro/news/free-lunch-for-llms-confidence-the-rl-critic-already-knows.md", "text": "https://wpnews.pro/news/free-lunch-for-llms-confidence-the-rl-critic-already-knows.txt", "jsonld": "https://wpnews.pro/news/free-lunch-for-llms-confidence-the-rl-critic-already-knows.jsonld"}}