Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media

A new audit framework, the Cross-Platform Fairness Evaluation (CPFE), applied to four transformer models (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on a Kaggle mental health corpus (n=35,556) found consistent cross-platform AUC degradation of 30.3-35.4% on Reddit (n=6,257) and 37.9-39.5% on Twitter (n=2,883) relative to within-platform performance (AUC 0.983-0.987), with calibration error rising from 0.056-0.060 in-domain to 0.196-0.229 on Reddit and 0.499-0.542 on Twitter. The study, released on arXiv (2608.26138v1), recommends cross-platform validation across all five CPFE axes as a standard requirement for mental health NLP systems.

arXiv:2608.26138v1 Announce Type: new Abstract: We introduce the Cross-Platform Fairness Evaluation CPFE framework -- a five-axis audit protocol covering discriminative performance, calibration, statistical significance, prediction equity, and attribution stability -- and apply it to four transformer models BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa trained on a Kaggle mental health corpus n=35,556 and evaluated on Reddit n=6,257 and Twitter n=2,883 test sets with emotion labels mapped to clinical proxies. All three independently evaluated models exhibit consistent and substantial cross-platform AUC degradation 30.3-35.4% on Reddit, 37.9-39.5% on Twitter relative to within-platform performance AUC 0.983-0.987 , confirmed across five independent training seeds. Calibration failure is concurrent and severe: ECE rises from 0.056-0.060 in-domain to 0.196-0.229 on Reddit and 0.499-0.542 on Twitter. Platform-specific temperature scaling reduces mean ECE by 88.0% without altering discriminative performance mean |delta AUC|<0.01 , confirming separable failure modes. Prediction equity analysis reveals large cross-platform disparities raw DI < 0.17; prior-shift-adjusted DI: 0.11-0.29 on Reddit , with equalized odds differences of 0.753-0.830 for mental health proxy classes on Reddit and 0.755-0.831 for anxiety on Twitter. Attribution stability analysis shows near-complete vocabulary divergence across platforms Jaccard J=0 in 14/16 model-class pairs at K=10 . These findings support treating cross-platform validation across all five CPFE axes as a standard requirement for mental health NLP systems in heterogeneous environments. In a single-seed fine-tuning experiment, mean AUC improved by 0.216, suggesting target-platform labels provide greater benefit as training signal than as calibration signal.