{"slug": "behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge", "title": "Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals", "summary": "A study posted to arXiv (2610.30512v1) found that rerunning the default Arena-Hard v2.0 setup with GPT-4.1 as judge at temperature zero moved the aggregate score by about one point but flipped the win/tie/loss outcome on 21.2% of 500 prompts. Varying the judge's instructions changed outcomes on up to 28.2 percentage points more prompts than unchanged reruns did on average, with the aggregate score falling 4.09 points in one setup, while an accuracy-prioritizing setup shifted outcomes on 6.7 percentage points more prompts with almost no aggregate movement. The authors conclude a stable aggregate score does not establish that individual verdicts are repeatable, and recommend evaluations relying on individual verdicts repeat each setup and report both aggregate scores and prompt-level agreement.", "body_md": "arXiv:2610.30512v1 Announce Type: new \nAbstract: LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement.", "url": "https://wpnews.pro/news/behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge", "canonical_source": "https://arxiv.org/abs/2610.30512", "published_at": "2026-10-23 04:00:00+00:00", "updated_at": "2026-10-23 04:12:58.264950+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety", "artificial-intelligence"], "entities": ["arXiv", "Arena-Hard v2.0", "GPT-4.1", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge", "markdown": "https://wpnews.pro/news/behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge.md", "text": "https://wpnews.pro/news/behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge.txt", "jsonld": "https://wpnews.pro/news/behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge.jsonld"}}