Position: Fairness Failure in Generative Models is an Evaluation Problem

A new position paper from arXiv (2608.16974v1) argues that fairness failures in generative models are fundamentally an evaluation problem, citing a lack of comparability and actionability in current fairness findings. The authors propose Fairness Cards, a standardized reporting artifact that makes evaluation choices explicit, to improve reproducibility and accountability in generative AI fairness assessment.

arXiv:2608.16974v1 Announce Type: new Abstract: Despite groundbreaking advancements in generative models during the last decade, concerns about their lack of fairness, reinforcing societal inequalities and harming marginalized groups, remain under-addressed and difficult to act upon. This position paper argues that fairness failures in generative models, albeit driven by multiple factors, are ultimately stemming from an evaluation problem: fairness findings are rarely comparable across papers or actionable for deployment decisions. This paper diagnoses recurring empirical and conceptual failure modes in current practice and motivates a shift from ad-hoc bias checks to standardized, generative-specific evaluation. We propose Fairness Cards as a minimal reporting artifact that makes evaluation choices explicit prompt families, counterfactual protocols, metrics, and refusal handling enabling reproducibility, comparability, and accountability. We conclude with additional recommendations towards a paradigm shift in evaluation standards. Our project page can be found at https://mariiavladimirova.github.io/fairness-cards .