{"slug": "agora-reliability-aware-aggregation-of-llm-judges-via-item-response-theory", "title": "AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory", "summary": "Researchers posted arXiv:2611.05556v1, introducing AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator for combining multiple LLM judges that learns each judge's reliability from all items jointly without human labels. Across three benchmarks, AGORA matched or exceeded the Dawid-Skene confusion-table method on large evaluation sets, and its advantage widened as evaluation sets shrank toward a few hundred items, remaining accurate as judge pools grew more uneven. A label-free dispersion statistic flags in advance when trust weighting helps, and AGORA's learned trust weights track per-judge human validity.", "body_md": "arXiv:2611.05556v1 Announce Type: new \nAbstract: LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity.", "url": "https://wpnews.pro/news/agora-reliability-aware-aggregation-of-llm-judges-via-item-response-theory", "canonical_source": "https://arxiv.org/abs/2611.05556", "published_at": "2026-11-05 05:00:00+00:00", "updated_at": "2026-11-05 05:21:48.864082+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["AGORA", "Dawid-Skene", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/agora-reliability-aware-aggregation-of-llm-judges-via-item-response-theory", "markdown": "https://wpnews.pro/news/agora-reliability-aware-aggregation-of-llm-judges-via-item-response-theory.md", "text": "https://wpnews.pro/news/agora-reliability-aware-aggregation-of-llm-judges-via-item-response-theory.txt", "jsonld": "https://wpnews.pro/news/agora-reliability-aware-aggregation-of-llm-judges-via-item-response-theory.jsonld"}}