Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

A new arXiv paper (2610.19166v1) proposes a regression-based meta-evaluation of text generation metrics that learns on calibrated metric outputs to predict human ratings, grounded against a trivial constant-mean baseline and human agreements. The authors built a large-scale multi-lingual style-personalized text generation benchmark with multiple annotations across 10 topologically diverse languages and four evaluation dimensions, then meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over 40 evaluation settings. The analysis found that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings, and identified human-annotation characteristics that determine metric performance plus conditions under which an ensemble outperforms its individual constituents.

arXiv:2610.19166v1 Announce Type: new Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation SPTG benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms string-based, embedding-based, and LLM-judges and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.