Your Agent's Confidence Score Is Not a Probability

A developer warns that self-reported confidence scores from AI agents are not calibrated probabilities and should not be used for gating decisions. The developer argues that these scores are generated by the same model that produced the output, making them circular and unreliable, and recommends routing based on independent checks instead.

Ask an agent how sure it is and it will happily tell you. "Confidence: 0.92." It looks like a probability. It renders nicely in a dashboard. Teams wire it into routing logic: high confidence, auto-approve; low confidence, send to a human. It feels rigorous. It is not rigorous. A self-reported confidence score is the agent grading its own homework, and it is one of the most seductive false signals in production agentic systems. If you are gating anything on it, you are trusting the defendant's opinion of their own alibi. When an LLM emits confidence: 0.92 , that number is not a calibrated posterior. It is another token sequence, generated by the same forward pass that produced the answer you are unsure about. It shares a substrate with the output it is describing. If the model hallucinated a file path, the same weights that invented the path will cheerfully assign it 0.9 confidence, because from the inside, a confident fabrication and a confident fact are indistinguishable. This is not a prompt-engineering problem you can fix with "be honest about your uncertainty." You can push the distribution around, but you cannot make a model's self-report into independent evidence, because there is no independent ground truth in the loop. The grader and the graded are the same network. This is exactly why, when we build evals at agent-eval https://github.com/ , we rank evidence on an independence axis — independent to corruptible — not a cost axis of cheap to expensive. Three tiers: A self-reported confidence score is not even Tier 3. Tier 3 at least lets a separate model inspect an artifact. Self-confidence is the judged model judging itself in the same breath — maximally circular. It belongs at the very corruptible end of the axis. Tier 1+2 are the real-time gate. They are deterministic, roughly free, and fast, so they can sit in the hot path and block a bad run before it ships. Tier 3 is offline-only: metered, slow, non-deterministic. A judge cannot live in your latency budget. And self-reported confidence, despite looking cheap enough to gate on, is corruptible enough that gating on it is worse than gating on nothing — because it gives you false comfort. A model judging another model's reasoning is circular. Tier 1+2 can run over agent trajectories. Tier 3 cannot — if you let a judge grade the reasoning steps, judge and judged share a substrate and there is no independent ground truth. So Tier 3 may only inspect artifacts the judged agent didn't get to write. Self-confidence violates this rule harder than anything else: it is a claim about the reasoning, authored by the reasoner. Don't route on the agent's opinion of itself. Route on independent checks. Here is the shape of it in TypeScript: type AgentOutput = { answer: string; filePath: string; selfConfidence: number; // present, and deliberately ignored for gating }; type GateResult = { pass: boolean; tier: 1 | 2; reason: string }; async function gate out: AgentOutput, task: string : Promise<GateResult { // Tier 1: unforgeable proof if out.answer.trim return { pass: false, tier: 1, reason: "empty" }; if await fileExists out.filePath return { pass: false, tier: 1, reason: "hallucinated path" }; // Tier 2: statistical signal vs a baseline the agent didn't author const sim = await cosineSim embed out.answer , embed task ; if sim < 0.55 return { pass: false, tier: 2, reason: off-task sim=${sim.toFixed 2 } }; // out.selfConfidence is never consulted. It's the defendant's alibi. return { pass: true, tier: 2, reason: "cleared independent checks" }; } Notice selfConfidence is present in the type and never read in the gate. That is the point. You can log it, correlate it against outcomes offline, even discover it's anti-correlated with correctness — but it does not get a vote in the hot path. Most production failures are boring and mechanical: stale data, a crash, malformed output, a hallucinated path, an empty result. Every one of those is caught at Tier 1+2 alone, deterministically, for about zero dollars. Reserve the model-as-judge for the roughly 20% subjective tail — tone, helpfulness, whether an explanation is actually clear — and label its output honestly as "opinion, not evidence." A judge that says 7/10 is a suggestion for a human, not a gate. All of this assumes you can actually inspect what the agent did — the real inputs after resolution, the real tool outputs, the real intermediate steps. That is the other half of the workflow. AgentLens https://github.com/ captures the trace : every model and tool step, resolved inputs, raw outputs. agent-eval scores and gates the output ; AgentLens gives you the unforgeable, agent-didn't-author trace data for Tier 1+2 to score against — and the debugging surface for when a gate goes red and you need to know why . The pairing matters here specifically because the whole failure mode of this post is trusting the agent's narration of itself. AgentLens replaces the narration with the record. agent-eval judges the record, not the narration. Self-reported confidence is the narration. Stop routing on it. Route on proof.