Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables

Researchers proposed counterfactual (CF) marginalisation, a test-time evaluation procedure for assessing how robust classification models are to nuisance variables such as age or sex, according to a paper published as arXiv:2609.10778v1. The method uses a CF image generator to create counterfactual versions of each test image, averages predictions over a target intervention distribution, and defines metrics for CF risk, calibration, stability and worst-case sensitivity. The authors demonstrate the framework's utility for quantitative robustness evaluation.

arXiv:2609.10778v1 Announce Type: new Abstract: Machine learning models can achieve strong test performance while relying on demographic or acquisition-related shortcuts. We propose counterfactual CF marginalisation as a test-time evaluation procedure for assessing robustness of classification models to such variables. Given a CF image generator, we intervene on nuisance parent variables such as age or sex, generate CF versions of each test image, and average predictions over a target intervention distribution. This produces intervention-aware predictions that marginalise demographic effects while preserving patient-specific latent information. We use these predictions to define metrics for CF risk, calibration, stability and worst-case sensitivity. We demonstrate this framework's utility for quantitative robustness evaluation.