Focal Calibration Loss: Controlling Posterior Distortion in Deep Neural Classifiers

Researchers studying Focal Calibration Loss (FCL) found that while it preserves the Bayes decision rule for all gamma and lambda, it is generally not proper for gamma > 0, causing the Bayes-optimal probability vector to deviate from the true posterior. The anchor term controls this distortion, with bounds of sqrt(log K / lambda) and O(1/lambda) for interior posteriors, and experiments on CIFAR-10/100, Tiny-ImageNet, text, and medical multi-label tasks show FCL is competitive but not dominant, with regime-dependent performance.

arXiv:2410.18321v3 Announce Type: replace Abstract: Confidence calibration matters wherever a classifier's probabilities, not just its labels, are consumed downstream. We study Focal Calibration Loss FCL , which adds a squared probability-error multiclass Brier anchor to the focal objective, $\mathcal{L}{\mathrm{FCL}}^{\gamma,\lambda} = \mathcal{L}{\mathrm{focal}}^{\gamma} + \lambda |\hat{p} x - e y| 2^2$. Our analysis separates two properties that are easily conflated: FCL is classification-calibrated for every $\gamma, \lambda \ge 0$, preserving the Bayes decision rule, yet for $\gamma 0$ it is generally not proper, so its Bayes-optimal probability vector is displaced from the true posterior. The main result quantifies that displacement and shows the anchor controls it: bounded by $\sqrt{\log K / \lambda}$ for every posterior and minimizer without regularity assumptions, improving to $O 1/\lambda $ for interior posteriors, with an exact first-order expansion identifying the bias and corresponding population $\ell 2$ calibration guarantees. We verify these population statements directly, minimizing the conditional risk on the simplex with no network involved: the posterior-distortion rate matches its prediction to a median fitted slope of $-0.994$, and exact population squared calibration error follows the predicted $\lambda^{-2}$ law slopes $\approx -1.99$ . Across CIFAR-10/100, Tiny-ImageNet, text and medical multi-label tasks, FCL is competitive rather than dominant, and the picture is regime- and metric-dependent: under a common validation-split protocol the validation-adaptive AdaFocal attains lower binned calibration error, while FCL attains lower NLL, Brier and error on two of three settings. On transformers its calibration advantage is absent, and a from-scratch experiment tested and did not support the conjecture that pretraining explains this. We report both the gains and the failure regimes.