{"slug": "a-consensus-based-framework-for-relative-preference-evaluation-of-large-language", "title": "A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models", "summary": "A new consensus-based evaluation framework for large language models (LLMs) measures relative preference among model-generated responses rather than absolute correctness, according to a paper on arXiv. The framework uses a panel of five state-of-the-art LLMs to rank anonymized candidate responses, producing a Relative Intelligence Index (RII) that reflects how frequently a model's responses are preferred by other models. The study found consistent preference patterns across domains including programming, general knowledge, safety, logical reasoning, and mathematics, though the authors caution that results reflect inter-model preference alignment rather than objective correctness or human judgment.", "body_md": "arXiv:2607.21632v1 Announce Type: new\nAbstract: Traditional benchmarks for LLMs primarily rely on static datasets and objective scoring metrics, which often fail to capture differences in response quality when multiple answers are acceptable. In such settings, correctness alone is insufficient to distinguish between responses that vary in clarity, completeness, and usefulness.\nThis paper introduces a consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness. Instead of evaluating outputs against a fixed ground truth, we assess how a panel of diverse LLMs ranks anonymized candidate responses to the same prompt. This approach treats aggregate inter-model agreement as a proxy for perceived response quality under blind conditions.\nWe conduct a controlled study using five state-of-the-art LLMs across multiple domains, including programming, general knowledge, safety, logical reasoning, and mathematics. Each model generates responses and independently ranks peer outputs through a structured voting process. Scores are aggregated into a Relative Intelligence Index (RII), representing how frequently a model's responses are preferred by other models.\nOur findings reveal consistent preference patterns across domains, with certain models more frequently ranked highly by their peers. However, we emphasize that these results reflect inter-model preference alignment rather than objective correctness or human judgment. This framework provides a scalable, model-driven method for comparative evaluation, offering an alternative perspective on response quality in scenarios where multiple valid answers exist. While not directly aligned with human evaluation, prior work suggests that aggregated model preferences can partially correlate with human judgments, motivating this as a proxy signal.", "url": "https://wpnews.pro/news/a-consensus-based-framework-for-relative-preference-evaluation-of-large-language", "canonical_source": "https://arxiv.org/abs/2607.21632", "published_at": "2026-07-27 04:00:00+00:00", "updated_at": "2026-07-27 04:24:30.633402+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/a-consensus-based-framework-for-relative-preference-evaluation-of-large-language", "markdown": "https://wpnews.pro/news/a-consensus-based-framework-for-relative-preference-evaluation-of-large-language.md", "text": "https://wpnews.pro/news/a-consensus-based-framework-for-relative-preference-evaluation-of-large-language.txt", "jsonld": "https://wpnews.pro/news/a-consensus-based-framework-for-relative-preference-evaluation-of-large-language.jsonld"}}