A Canary in the Neural Mine: Catching an Image Classifier's Silent, Confident Failures

A new arXiv paper (2610.33063v1) introduces CANARY, a training-free failure detector that matches or improves Maximum Softmax Probability (MSP) on 382 of 385 shift cells measured by e-AURC and 379 of 385 on AUROC across five backbones, with worst-case regression within seed noise at -0.0002 AUROC. CANARY computes a per-class kNN distance on the PCA residual of the penultimate feature, percentile-calibrates it on validation data, and fuses it with MSP through a single validation-tuned weight, with every parameter selected on in-distribution data alone. The authors report the largest gains on natural distribution shifts (ImageNet-A/R, ObjectNet) and describe a headroom law in which gains grow as baseline MSP quality falls, letting a single clean-validation number forecast the improvement.

arXiv:2610.33063v1 Announce Type: new Abstract: Deployed vision models auto-accept confident predictions, so high-confidence errors propagate silently. Maximum Softmax Probability MSP , the standard failure predictor, has a structural blind spot: as a scalar read from the classifier head, it is first-order blind to feature variation orthogonal to its confidence gradient. Through a factorial analysis of feature-space detectors, we identify a conditional-redundancy principle explaining why they fail to complement MSP: the class-separability geometry they measure is already encoded in MSP and collapses under fusion. The signal that survives is class-conditional typicality on the off-manifold residual---and because this residual's distribution varies by backbone, from near-Gaussian to heavy-tailed, it must be estimated non-parametrically. We instantiate these findings as CANARY, a training-free detector that computes a per-class kNN distance on the PCA residual of the penultimate feature, percentile-calibrates it on validation data, and fuses it with MSP through a single validation-tuned weight. CANARY's benefit follows a headroom law: gains grow as baseline MSP quality falls, so a single clean-validation number forecasts the improvement. With every parameter selected on in-distribution data alone, CANARY matches or improves 382 of 385 shift cells e-AURC; 379 on AUROC across five backbones, with worst-case regression within seed noise $-0.0002$ AUROC and the largest gains on natural distribution shifts ImageNet-A/R, ObjectNet ---where deployed models are most exposed.