{"slug": "fantastic-metrics-and-how-to-find-them-meta-evaluating-metrics-for-multi-lingual", "title": "Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation", "summary": "A new arXiv paper (2610.19166v1) proposes a regression-based meta-evaluation of text generation metrics that learns on calibrated metric outputs to predict human ratings, grounded against a trivial constant-mean baseline and human agreements. The authors built a large-scale multi-lingual style-personalized text generation benchmark with multiple annotations across 10 topologically diverse languages and four evaluation dimensions, then meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over 40 evaluation settings. The analysis found that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings, and identified human-annotation characteristics that determine metric performance plus conditions under which an ensemble outperforms its individual constituents.", "body_md": "arXiv:2610.19166v1 Announce Type: new \nAbstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.", "url": "https://wpnews.pro/news/fantastic-metrics-and-how-to-find-them-meta-evaluating-metrics-for-multi-lingual", "canonical_source": "https://arxiv.org/abs/2610.19166", "published_at": "2026-10-15 04:00:00+00:00", "updated_at": "2026-10-15 05:55:19.057023+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["arXiv", "LLM-judges"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fantastic-metrics-and-how-to-find-them-meta-evaluating-metrics-for-multi-lingual", "markdown": "https://wpnews.pro/news/fantastic-metrics-and-how-to-find-them-meta-evaluating-metrics-for-multi-lingual.md", "text": "https://wpnews.pro/news/fantastic-metrics-and-how-to-find-them-meta-evaluating-metrics-for-multi-lingual.txt", "jsonld": "https://wpnews.pro/news/fantastic-metrics-and-how-to-find-them-meta-evaluating-metrics-for-multi-lingual.jsonld"}}