RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs

Researchers introduced the Relative Probability Association Metric (RPAM), a new upstream metric for evaluating associations in generative language models. Testing on Mistral-7B-Instruct, Mistral-7B, and GPT-2, RPAM showed strong predictive validity for real-world associations and downstream biases, outperforming prior metrics.

arXiv:2607.05679v1 Announce Type: new Abstract: Language models LMs exhibit problematic biases, such as stereotypes. Effectively analyzing and mitigating such biases requires accurate and generalizable evaluation methods of the underlying associations. Some existing approaches focus on downstream metrics that analyze associations in generated text. Since generated text content can vary drastically across LMs, such metrics often require specialized evaluation datasets, which limits the generalization of such downstream metrics. In contrast, upstream metrics examine LMs at the fundamental level of embeddings or continuation probabilities, enabling principled association analyses across LMs. Yet, to date, no upstream metric for generative LMs has uncovered a strong relationship with real-world associations, including those measured in generated text. To address this gap, we introduce the Relative Probability Association Metric RPAM , an association evaluation metric for generative LMs. For three LMs of different quality of language generation and purpose Mistral-7B-Instruct, Mistral-7B, and GPT-2 and well-studied evaluation datasets WEAT-WS, Bellezza, WS-353, and SST2 , we find a strong relationship between upstream RPAM measurements and corresponding implicit and explicit associations observed in humans, as well as biases measured downstream with LM-specific tasks, outperforming prior record values where applicable.