{"slug": "llm-as-judge-done-right", "title": "LLM-as-Judge Done Right", "summary": "A tutorial on LLM-as-judge evaluation recommends splitting grading into narrow single-criterion judges with binary or low-cardinality rubrics rather than one broad 1–10 quality score, and calibrating each judge against human labels before trusting it. The lesson, part of an agentic AI course following a prerequisite on deterministic graders and assertions, argues that four separate judges for factual accuracy, groundedness, resolution clarity and tone are more reliable than one broad judge and identify what failed when one does. It also advises showing the judge only the evidence needed for its criterion and requiring reasoning before the verdict in structured output.", "body_md": "· Agentic AI · 9 min read\n\n### 📋 Prerequisites\n\n- Deterministic Graders and Assertions (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Decide when an LLM judge is the right grader and when it is not\n- Write analytic rubrics with binary or low-cardinality criteria that judges apply consistently\n- Choose between pointwise, pairwise, and reference-guided judging\n- Measure and mitigate position, verbosity, and self-preference bias\n- Calibrate a judge against human labels using agreement metrics\n\n## What Judges Are For\n\nAfter Lesson 5, your deterministic graders handle outcomes, policies, schemas, and budgets. What’s left is the stuff code can’t check:\n\n- Is the response **accurate** relative to what the tools returned?\n- Is it **grounded** , or does it invent policy?\n- Is it **clear and appropriate** in tone for an upset customer?\n- Did the agent **ask for clarification** when the request was genuinely ambiguous?\n- Is the reasoning in a long trajectory **sound** , even when the outcome was right?\n\nThese are judgment calls. An LLM judge can make them at scale, if you design the judge carefully and **prove** that it agrees with humans. An unvalidated judge is a random number generator with excellent grammar.\n\n## Anatomy of a Good Judge\n\nA judge has four parts:\n\n1. **The criterion** : one specific property, not “overall quality”\n2. **The evidence** : exactly what the judge needs to see (the user message, tool outputs, the response), and nothing more\n3. **The rubric** : an explicit decision procedure with low-cardinality outputs\n4. **The output format** : structured, with reasoning*before* the verdict\n\n### One Criterion per Judge\n\n“Rate the response 1–10 for quality” is the most common judge prompt and the least useful. The judge silently weighs a dozen things differently on every call. Split it:\n\n| Criterion | Question | \n|---|---|\n| Factual accuracy | Does every factual claim match the tool outputs? | \n| Groundedness | Does it state any policy or promise not supported by the KB? | \n| Resolution clarity | Would the customer know exactly what happened and what happens next? | \n| Tone | Is it appropriate for the customer’s emotional state? | \n\nFour narrow judges are more reliable than one broad one, and when one fails you know *what* failed.\n\n### Binary or Low-Cardinality Scales\n\nJudges are much more consistent on **pass/fail** or **3-point** scales than on 1–10. The difference between a 6 and a 7 isn’t something a model (or a human) applies consistently. If you need gradation, use an **analytic rubric**: several binary checks that you sum.\n\n```\nCriterion: Factual accuracy\n\nAnswer each question YES or NO based only on the TOOL OUTPUTS and RESPONSE.\nQ1. Does every amount mentioned in the response appear in the tool outputs?\nQ2. Does every invoice or date mentioned match the tool outputs?\nQ3. If the response says an action was taken, does a tool output confirm it succeeded?\nQ4. Does the response avoid claiming facts about the account that no tool returned?\n\nPASS only if all four are YES.\n```\n\nThis is far more reproducible than “rate accuracy 1–5,” and the per-question answers double as a diagnostic.\n\n### Show Only the Evidence Needed\n\nGive the judge the inputs it needs to decide the criterion, not the entire trace. More context means more distraction and more cost. For factual accuracy: the user message, the tool outputs, and the response. For tone: the user message and the response. The system prompt usually isn’t needed, and including it can bias the judge toward the agent’s own framing.\n\n### Reasoning Before Verdict\n\nAsk for reasoning first, then the verdict, in structured output. Verdict-first outputs get rationalized after the fact.\n\n## Implementing a Judge Grader\n\n``` python\nimport json\nfrom dataclasses import dataclass\n\nJUDGE_TEMPLATE = \"\"\"You are grading a customer-support agent's reply for ONE criterion.\n\n## Criterion\n{criterion}\n\n## Rubric\n{rubric}\n\n## Evidence\n<user_message>\n{user_message}\n</user_message>\n\n<tool_outputs>\n{tool_outputs}\n</tool_outputs>\n\n<agent_response>\n{response}\n</agent_response>\n\n{case_notes}\n\nRespond with JSON only:\n{{\"answers\": {{\"Q1\": \"YES|NO\", ...}}, \"reasoning\": \"<brief>\", \"verdict\": \"PASS|FAIL\"}}\n\"\"\"\n\n@dataclass\nclass RubricJudge:\n    name: str\n    criterion: str\n    rubric: str\n    judge_llm: callable        # (prompt: str) -> str, temperature 0\n    include_tools: bool = True\n\n    def grade(self, case, run) -> Score:\n        tool_outputs = \"\\n\".join(\n            f\"{s.name}({json.dumps(s.input)}) -> {json.dumps(s.output)}\"\n            for s in run.trace.of_kind(\"tool\")\n        ) if self.include_tools else \"(not shown)\"\n        notes = case.expected.get(\"rubric_notes\", \"\")\n        prompt = JUDGE_TEMPLATE.format(\n            criterion=self.criterion, rubric=self.rubric,\n            user_message=case.input, tool_outputs=tool_outputs,\n            response=run.response,\n            case_notes=f\"## Case-specific notes\\n{notes}\" if notes else \"\",\n        )\n        raw = self.judge_llm(prompt)\n        try:\n            out = json.loads(extract_json(raw))\n        except ValueError:\n            return Score(self.name, 0.0, False, \"judge returned unparseable output\",\n                         {\"raw\": raw})\n        yes = sum(v == \"YES\" for v in out.get(\"answers\", {}).values())\n        total = max(len(out.get(\"answers\", {})), 1)\n        passed = out.get(\"verdict\") == \"PASS\"\n        return Score(self.name, yes / total, passed, out.get(\"reasoning\", \"\"),\n                     {\"answers\": out.get(\"answers\")})\n```\n\nNote the `rubric_notes` hook from Lesson 4: case authors can add per-case guidance (“the customer explicitly asked not to be escalated”) without changing the judge.\n\n## Pointwise, Pairwise, and Reference-Guided\n\n| Mode | How it works | Best for | \n|---|---|---|\n| **Pointwise** | Judge one response against a rubric | Absolute quality gates; monitoring | \n| **Pairwise** | Judge which of two responses is better | Comparing two agent versions; subtle quality differences | \n| **Reference-guided** | Judge against an expert reference answer | Factual tasks where an ideal answer exists | \n\n**Pairwise** is more sensitive than pointwise for small differences. Humans and models are both better at “which is better?” than “how good is this?” Use it when evaluating whether a prompt change improved tone or clarity. Report it as a **win rate** with ties allowed.\n\n**Reference-guided** judging helps when a correct answer exists but can be phrased many ways. Tell the judge the reference is *one* acceptable answer, not *the* answer, or it will penalize valid alternatives.\n\n## Known Judge Biases\n\nThese are well documented in the research literature and show up in practice. Test for each one.\n\n**Position bias.** In pairwise mode, judges prefer whichever response appears first (or second, depending on the model). *Mitigation:* run every pair in both orders. Count a win only if it holds in both; otherwise it’s a tie.\n\n``` php\ndef pairwise(judge, case, run_a, run_b) -> str:\n    v1 = judge.compare(case, first=run_a, second=run_b)   # \"first\" | \"second\" | \"tie\"\n    v2 = judge.compare(case, first=run_b, second=run_a)\n    if v1 == \"first\" and v2 == \"second\": return \"A\"\n    if v1 == \"second\" and v2 == \"first\": return \"B\"\n    return \"tie\"\n```\n\n**Verbosity bias.** Judges favor longer responses, even when the extra length adds nothing. *Mitigation:* put concision in the rubric explicitly; check the correlation between judge score and response length on your data. If it’s strong and humans don’t share it, the judge is biased.\n\n**Self-preference bias.** Models tend to rate outputs from their own family higher. *Mitigation:* where possible, use a judge from a different model family than the agent, or at least validate that the judge doesn’t systematically prefer one version for reasons humans don’t.\n\n**Sycophancy toward confident text.** Assertive, well-formatted wrong answers get rated higher than hedged correct ones. *Mitigation:* give the judge the tool outputs so it can check facts, instead of asking it to assess plausibility.\n\n**Rubric drift across model versions.** Upgrade the judge model, and your scores shift even though the agent didn’t change. *Mitigation:* pin the judge model version, and treat a judge upgrade like an agent change: recalibrate before adopting.\n\n## Calibration: Proving the Judge Works\n\nThis is the step most teams skip, and it’s the one that makes the judge’s numbers mean anything.\n\n### Step 1: Collect Human Labels\n\nTake 100–200 runs that span the range of quality: clear passes, clear fails, and borderline cases. Have domain experts label them using **the same rubric** the judge uses. Two labelers per item if you can afford it, so you can measure human-human agreement too.\n\n### Step 2: Measure Agreement\n\n``` python\nfrom sklearn.metrics import cohen_kappa_score, confusion_matrix\n\ndef calibration_report(human: list[bool], judge: list[bool]) -> dict:\n    tn, fp, fn, tp = confusion_matrix(human, judge, labels=[False, True]).ravel()\n    return {\n        \"agreement\": (tp + tn) / len(human),\n        \"cohen_kappa\": cohen_kappa_score(human, judge),\n        \"judge_false_pass\": fp / (fp + tn) if fp + tn else 0.0,   # judge PASS, human FAIL\n        \"judge_false_fail\": fn / (fn + tp) if fn + tp else 0.0,   # judge FAIL, human PASS\n    }\n```\n\nRaw agreement is misleading when most items pass. If 90% of responses are good, a judge that always says PASS gets 90% agreement. **Cohen’s kappa** corrects for chance. Rough guide:\n\n| Kappa | Interpretation | \n|---|---|\n| below 0.4 | Don’t use this judge for decisions | \n| 0.4–0.6 | Moderate; useful for trends, not gates | \n| 0.6–0.8 | Substantial; usable for most purposes | \n| above 0.8 | Excellent; comparable to good human-human agreement | \n\nAlso compare to **human-human** kappa. If two experts only agree at 0.65, a judge at 0.62 is about as good as you’ll get, and the rubric (not the judge) may need clarifying.\n\n### Step 3: Read the Disagreements\n\nThe confusion matrix tells you how often the judge is wrong. Reading the disagreements tells you *why*. Typical findings:\n\n- The rubric was ambiguous, and humans interpreted it differently from the judge. Fix the rubric.\n- The judge lacked evidence (e.g., couldn’t see a tool output). Fix the evidence.\n- The judge has a systematic bias (verbosity, leniency). Add explicit rubric language or few-shot examples of the boundary.\n\nIterate the rubric, rerun on the labeled set, and recompute kappa. Hold out a portion of the labels so you don’t overfit the rubric to them.\n\n### Step 4: Decide What the Judge Is Allowed to Do\n\nAsymmetric errors matter. For a safety-relevant criterion, a judge with a 10% false-pass rate is dangerous; for tone, it’s fine. Write down, per judge:\n\n- Is it a **gate** or**weighted** ?\n- What kappa and false-pass rate did it achieve, on which label set, with which judge model version?\n- When must it be recalibrated? (On any judge-model change, rubric change, or quarterly.)\n\n## Cost and Practicality\n\n- **Use a smaller judge where it calibrates well.** A small, fast model often matches a large one on narrow binary criteria. Calibration tells you whether it does.\n- **Cache judge verdicts** keyed on (criterion version, judge model, evidence hash). Rerunning a report shouldn’t re-pay for identical judgments.\n- **Run judges only on runs that passed the gates.** If the agent refunded the wrong invoice, there’s no point grading its tone.\n- **Sample for expensive judges.** For monitoring, judging a random 10% of traffic is often enough to detect trends (Lesson 11).\n\n## Key Takeaways\n\n- Use judges only for what code can’t check: accuracy against evidence, groundedness, clarity, tone\n- One criterion per judge; binary or analytic rubrics; reasoning before verdict; only the evidence needed\n- **Pairwise** is more sensitive for comparisons; always run both orders to cancel position bias\n- Test for position, verbosity, and self-preference bias; pin the judge model version\n- **Calibrate against human labels** with Cohen’s kappa, read the disagreements, and document what each judge is trusted to decide\n\n## Exercise\n\n1. Write two rubric judges for your agent: factual accuracy (with tool outputs as evidence) and resolution clarity\n2. Label 60 runs yourself (or with a colleague) using the same rubrics\n3. Compute agreement, kappa, false-pass, and false-fail rates for each judge\n4. Read every disagreement, revise the rubric once, and recompute on 20 held-out labels\n5. Record each judge’s calibration results in a `JUDGES.md` file next to your graders\n\nIf either judge’s kappa stays below 0.5, don’t use it as a gate. Make it a weighted signal and keep improving the rubric.", "url": "https://wpnews.pro/news/llm-as-judge-done-right", "canonical_source": "https://superml.org/tutorials/agent-eval-llm-as-judge", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:16.358559+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "artificial-intelligence"], "entities": [], "also_reported_by": [], "takeaway": "Split evaluation into one narrow judge per criterion with binary or 3-point analytic rubrics (e.g. four YES/NO factual-accuracy checks) instead of a single 1–10 quality score, and calibrate each judge against human labels before trusting it.", "recommendation": {"advice": "Replace broad 1–10 quality prompts with one judge per criterion using binary or 3-point analytic rubrics, feed only the evidence that criterion needs, and require reasoning before the verdict.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/llm-as-judge-done-right", "markdown": "https://wpnews.pro/news/llm-as-judge-done-right.md", "text": "https://wpnews.pro/news/llm-as-judge-done-right.txt", "jsonld": "https://wpnews.pro/news/llm-as-judge-done-right.jsonld"}}