Judging an LLM judge?

A developer built an open-source tool called judgeDjudge to evaluate LLM-as-a-judge systems, testing judges for repeated-run consistency, position bias, sensitivity to verbosity, and accuracy in preferring the better response. The roughly 200-line evaluator uses a dataset of tasks, rubrics, ideal responses, and negative responses, and the developer is seeking feedback on additional failure modes and testing methods.

I have been reading some blog posts about LLM as a judge and was building a small evaluator to evaluate the judge itself . My method is simple: The dataset is: task rubric ideal response negative response The idea is then to test different models as judges for things like: repeated-run consistency position bias sensitivity to verbosity accuracy / ability to prefer the better response Here, “negative response” doesn’t necessarily mean a wrong answer. It can just be a response that is less preferred according to the rubric. I have an initial version with around 200 lines of code https://github.com/maylad31/judgeDjudge https://github.com/maylad31/judgeDjudge But I’m more interested in discussing the idea. If you have used LLM judges in practice, are there other failure modes or better ways of testing them? Happy to hear criticism or suggestions or positive things about my method/code.