When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

A large-scale study of 53 LLM runners generating 265,000 samples found that self-consistency and cross-model agreement are weak and regime-dependent proxies for correctness, not reliable confidence signals. The most consistent frontier models showed over-confidence, agreeing on 77% of GPQA case-result entries but being wrong on 48% of those. The findings challenge the assumption that agreement among LLM judges indicates accuracy.

arXiv:2607.08065v1 Announce Type: new Abstract: LLM-as-judge Zheng et al., 2023 is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles Verga et al., 2024 or "mixture-of-experts" Shazeer et al., 2017 panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME -- 265,000 samples. Using majority-correctness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor rho 0.20-0.59, all positive under item-clustered resampling whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst -- over-confident yet no more accurate -- for the most consistent frontier model agreement =0.8 on 77% of GPQA case-result entries, 48% of those wrong . An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginal-preserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the de-identified per-run rows and answer distributions.