# Day 08/90: Who Evaluates the Evaluator? When LLM Judges Get It Wrong > Source: > Published: 2026-10-25 19:40:37+00:00 You update a prompt, generate a new answer, and ask another LLM to evaluate it. _ The score goes from 7/10 to 9/10._ That looks like progress. But what actually improved: the answer’s correctness, its presentation, or its ability to impress the evaluator? For an AI engineer, the evaluation pipeline deserves the same attention as the application itself. LLM-as-a-Judge means using a language model to assess another model’s output. You provide a question, candidate answers, and evaluation criteria. You might also include source documents or a reference answer. The judge then assigns scores or selects the better answer. This helps scale evaluation, but the evaluator also has limitations. The paper *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena* investigates limitations including position bias, verbosity bias, and reasoning errors. These are useful failure modes to test in an evaluation pipeline. **Position bias:** The order of candidate answers can influence the judge’s preference. **Verbosity bias:** A longer answer can receive more credit even when its extra detail adds little value. **Reasoning errors:** The judge can accept a convincing explanation that contains a factual or logical mistake. A detailed evaluation explanation can help with inspection, but it does not establish that the judgment is correct. Consider this fictional policy: Customers can cancel within 14 days of purchase. The policy provides no exceptions. **Question:** Can I cancel my purchase after 20 days? **Candidate A:** “No. The policy allows cancellation within 14 days, so a cancellation after 20 days falls outside that window.” **Candidate B:** “Yes. Although the standard cancellation window is 14 days, customers can request an extension to 30 days by contacting support.” Candidate B sounds helpful, but it invents an exception. Now imagine these judge outputs: | Evaluation run | First answer shown | Second answer shown | Judge selects | |---|---|---|---| | Original order | Candidate A | Candidate B | Candidate A | | Reversed order | Candidate B | Candidate A | Candidate B | **This is a hypothetical illustration, not an experimental result.** The preferred answer changed even though the question, evidence, and candidate answers stayed the same. That flags order sensitivity. Notice that we track the underlying candidate, not its display position. Otherwise, swapping labels could make a consistent judgment look inconsistent. “Rate this answer from 1 to 10” leaves too much open to interpretation. Use separate dimensions: | Dimension | What to check | |---|---| | Correctness | Does the answer contain factual or logical errors? | | Groundedness | Are its claims supported by the supplied evidence? | | Completeness | Does it address the required parts of the question? | | Relevance | Does it stay focused on the request? | Separate scores help expose trade-offs. An answer can be relevant and readable while still making unsupported claims. For document-based questions, give the judge the relevant source text. For calculations, check results with code where possible. For structured outputs, validate the schema directly. Use model judgment for aspects that require interpretation, alongside checks that can verify specific properties precisely. Prepare 20 questions with one candidate pair per question. Evaluate each pair twice: once in each order. That produces **40 judgments across 20 paired comparisons**. Map every verdict back to one of three outcomes: Define the **order-change rate** as: **Pairs whose mapped outcome changes ÷ total pairs** Under this definition, A → B, A → Tie, and Tie → B all count as changes. Tie → Tie does not. You can also report **strict reversals** separately: only A → B or B → A. Keeping ties visible makes the metric easier to interpret. Build a small evaluation set containing correct, incorrect, incomplete, and unsupported answers. Have reviewers use the same rubric and supporting evidence, then inspect disagreements with the model judge. Human review also needs quality control. Ambiguous criteria or incorrect reference judgments can make agreement misleading. Resolve unclear cases before treating those labels as a reliable baseline. A judge can consistently prefer the same answer and still be wrong. Order consistency measures stability under one change. It does not measure factual accuracy by itself. Likewise, agreement with human reviewers is useful only to the extent that their judgments are dependable. These checks provide different pieces of evidence. No single score establishes that an evaluator is reliable across every task. Before treating a higher evaluation score as an improvement, inspect the rubric, supporting evidence, order sensitivity, and agreement with reliable human judgments. **When an LLM judges your application, the judge becomes another component you need to evaluate.** Do you use LLM-as-a-Judge in production, and how do you measure its agreement with human reviewers?