{"slug": "judging-an-llm-judge", "title": "Judging an LLM judge?", "summary": "A developer built an open-source tool called judgeDjudge to evaluate LLM-as-a-judge systems, testing judges for repeated-run consistency, position bias, sensitivity to verbosity, and accuracy in preferring the better response. The roughly 200-line evaluator uses a dataset of tasks, rubrics, ideal responses, and negative responses, and the developer is seeking feedback on additional failure modes and testing methods.", "body_md": "I have been reading some blog posts about LLM as a judge and was building a small evaluator to evaluate the judge itself .\n\nMy method is simple:\n\nThe dataset is:\n\n```\ntask\n\nrubric\n\nideal response\n\nnegative response\n```\n\nThe idea is then to test different models as judges for things like:\n\n```\nrepeated-run consistency\n\nposition bias\n\nsensitivity to verbosity\n\naccuracy / ability to prefer the better response\n```\n\nHere, “negative response” doesn’t necessarily mean a wrong answer. It can just be a response that is less preferred according to the rubric.\n\nI have an initial version with around 200 lines of code \n\n[https://github.com/maylad31/judgeDjudge](https://github.com/maylad31/judgeDjudge) \n\nBut I’m more interested in discussing the idea.\n\nIf you have used LLM judges in practice, are there other failure modes or better ways of testing them?\n\nHappy to hear criticism or suggestions or positive things about my method/code.", "url": "https://wpnews.pro/news/judging-an-llm-judge", "canonical_source": "https://dev.to/mayank_laddha_ml/judging-an-llm-judge-51p0", "published_at": "2026-09-19 07:09:55+00:00", "updated_at": "2026-09-19 07:54:33.127370+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-research", "developer-tools"], "entities": ["judgeDjudge", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/judging-an-llm-judge", "markdown": "https://wpnews.pro/news/judging-an-llm-judge.md", "text": "https://wpnews.pro/news/judging-an-llm-judge.txt", "jsonld": "https://wpnews.pro/news/judging-an-llm-judge.jsonld"}}