arXiv:2608.26138v1 Announce Type: new Abstract: We introduce the Cross-Platform Fairness Evaluation (CPFE) framework -- a five-axis audit protocol covering discriminative performance, calibration, statistical significance, prediction equity, and attribution stability -- and apply it to four transformer models (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on a Kaggle mental health corpus (n=35,556) and evaluated on Reddit (n=6,257) and Twitter (n=2,883) test sets with emotion labels mapped to clinical proxies. All three independently evaluated models exhibit consistent and substantial cross-platform AUC degradation (30.3-35.4% on Reddit, 37.9-39.5% on Twitter) relative to within-platform performance (AUC 0.983-0.987), confirmed across five independent training seeds. Calibration failure is concurrent and severe: ECE rises from 0.056-0.060 in-domain to 0.196-0.229 on Reddit and 0.499-0.542 on Twitter. Platform-specific temperature scaling reduces mean ECE by 88.0% without altering discriminative performance (mean |delta AUC|<0.01), confirming separable failure modes. Prediction equity analysis reveals large cross-platform disparities (raw DI < 0.17; prior-shift-adjusted DI: 0.11-0.29 on Reddit), with equalized odds differences of 0.753-0.830 for mental health proxy classes on Reddit and 0.755-0.831 for anxiety on Twitter. Attribution stability analysis shows near-complete vocabulary divergence across platforms (Jaccard J=0 in 14/16 model-class pairs at K=10). These findings support treating cross-platform validation across all five CPFE axes as a standard requirement for mental health NLP systems in heterogeneous environments. In a single-seed fine-tuning experiment, mean AUC improved by 0.216, suggesting target-platform labels provide greater benefit as training signal than as calibration signal.
When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systems