{"slug": "benchmarking-llm-as-a-judge-for-long-form-output-evaluation", "title": "Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation", "summary": "Researchers have introduced LongJudgeBench, a new benchmark designed to evaluate the reliability of large language models (LLMs) when used as judges for long-form outputs. The benchmark reveals a substantial reliability gap, showing that current LLM judges remain unstable across diverse scenarios and that rubrics or references are helpful but not always sufficient for accurate evaluation. This work addresses a critical challenge as LLMs are increasingly deployed for long-form generation, where existing meta-evaluation benchmarks focus primarily on short-form outputs.", "body_md": "# Computer Science > Computation and Language\n\n[Submitted on 1 Jun 2026 (\n\n[v1](https://arxiv.org/abs/2606.01629v1)), last revised 2 Jun 2026 (this version, v2)]# Title:Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation\n\n[View PDF](/pdf/2606.01629)\n\n[HTML (experimental)](https://arxiv.org/html/2606.01629v2)\n\nAbstract:As large language models (LLMs) are increasingly used for long-form generation, reliably evaluating long-form outputs has become a critical challenge. LLM-as-a-judge offers a scalable alternative to human evaluation, yet its reliability in long-form output evaluation remains underexamined: existing meta-evaluation benchmarks focus mainly on short-form outputs. Compared with short-form evaluation, long-form evaluation is not merely a matter of output length; it often requires judges to make more complex document-level assessments of overall organization, task-relevant coverage and depth, cross-section consistency, and scenario-specific quality criteria. In this work, we introduce LongJudgeBench, a comprehensive benchmark for evaluating LLM judges on long-form outputs across diverse real-world scenarios and judging protocols. We systematically evaluate a broad range of LLM judges, covering multiple base models and judging settings. Our results reveal a substantial reliability gap: current LLM judges remain unstable across scenarios, and rubrics or references are helpful but not always sufficient. We hope LongJudgeBench will support future research on more robust, context-aware, and human-aligned LLM-as-a-judge methods. Our code is available at[this https URL].\n\n## Submission history\n\nFrom: Junjie Chen [[view email](/show-email/b56a0611/2606.01629)]\n\n**Mon, 1 Jun 2026 03:25:34 UTC (325 KB)**\n\n[[v1]](/abs/2606.01629v1)**[v2]** Tue, 2 Jun 2026 07:49:40 UTC (327 KB)\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/benchmarking-llm-as-a-judge-for-long-form-output-evaluation", "canonical_source": "https://arxiv.org/abs/2606.01629", "published_at": "2026-06-03 04:48:08+00:00", "updated_at": "2026-06-03 05:17:52.321853+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "natural-language-processing", "ai-research", "generative-ai"], "entities": ["LongJudgeBench", "LLM-as-a-judge"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-llm-as-a-judge-for-long-form-output-evaluation", "markdown": "https://wpnews.pro/news/benchmarking-llm-as-a-judge-for-long-form-output-evaluation.md", "text": "https://wpnews.pro/news/benchmarking-llm-as-a-judge-for-long-form-output-evaluation.txt", "jsonld": "https://wpnews.pro/news/benchmarking-llm-as-a-judge-for-long-form-output-evaluation.jsonld"}}