Conformity Breaks Conformal Prediction

A new arXiv paper (2609.0445v1) shows that conformal prediction certificates, which guarantee 90% coverage when an LLM answers alone, drop to 74% coverage when the same model is exposed to unanimous wrong answers from peer agents, a phenomenon the authors call a 'score-mechanism shift.' The researchers demonstrate that this shift silently breaks conformal prediction in multi-agent LLM systems, with coverage on low-confidence items falling from 87% to 47% under targeted attacks, and standard conformal fixes fail because the question distribution is unchanged while the model's scoring behavior is altered.

arXiv:2609.04445v1 Announce Type: new Abstract: A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer. The question is unchanged; the model's score for the correct answer changes. We call this a score-mechanism shift: clean calibration certifies how the model scores answers alone, but not how it scores them under peer pressure. We show that this shift silently breaks conformal prediction in multi-agent LLM systems. Across open-weight models and multiple-choice QA tasks, coverage falls from a calibrated 90% to 74% under unanimous-wrong peers at the standard alpha = 0.10 operating point. The average hides a sharper failure: by targeting the low-confidence items the certificate still covers, an attacker nearly halves coverage on that subgroup, from 87% to 47%, while the monitored average remains much higher. The failure also reaches the decision layer: a system that should escalate when uncertain can instead become confident enough to act on the attacker's wrong answer. Standard conformal fixes do not solve the problem, because the question distribution has not changed; the model's scoring behavior has.