cd /news/ai-agents/llm-as-judge-done-right · home › topics › ai-agents › article
[ARTICLE · art-172296] src=superml.org ↗ pub= topic=ai-agents verified=true sentiment=· neutral

LLM-as-Judge Done Right

A tutorial on LLM-as-judge evaluation recommends splitting grading into narrow single-criterion judges with binary or low-cardinality rubrics rather than one broad 1–10 quality score, and calibrating each judge against human labels before trusting it. The lesson, part of an agentic AI course following a prerequisite on deterministic graders and assertions, argues that four separate judges for factual accuracy, groundedness, resolution clarity and tone are more reliable than one broad judge and identify what failed when one does. It also advises showing the judge only the evidence needed for its criterion and requiring reasoning before the verdict in structured output.

recommendation

Replace broad 1–10 quality prompts with one judge per criterion using binary or 3-point analytic rubrics, feed only the evidence that criterion needs, and require reasoning before the verdict.

more in playbook: Evals and testing for LLM apps
read10 min views2 publishedNov 18, 2026
LLM-as-Judge Done Right
Image: Superml (auto-discovered)

· Agentic AI · 9 min read

📋 Prerequisites

  • Deterministic Graders and Assertions (previous lesson)

🎯 What You'll Learn

  • Decide when an LLM judge is the right grader and when it is not
  • Write analytic rubrics with binary or low-cardinality criteria that judges apply consistently
  • Choose between pointwise, pairwise, and reference-guided judging
  • Measure and mitigate position, verbosity, and self-preference bias
  • Calibrate a judge against human labels using agreement metrics

What Judges Are For #

After Lesson 5, your deterministic graders handle outcomes, policies, schemas, and budgets. What’s left is the stuff code can’t check:

  • Is the response accurate relative to what the tools returned?
  • Is it grounded , or does it invent policy?
  • Is it clear and appropriate in tone for an upset customer?
  • Did the agent ask for clarification when the request was genuinely ambiguous?
  • Is the reasoning in a long trajectory sound , even when the outcome was right?

These are judgment calls. An LLM judge can make them at scale, if you design the judge carefully and prove that it agrees with humans. An unvalidated judge is a random number generator with excellent grammar.

Anatomy of a Good Judge #

A judge has four parts:

  1. The criterion : one specific property, not “overall quality”
  2. The evidence : exactly what the judge needs to see (the user message, tool outputs, the response), and nothing more
  3. The rubric : an explicit decision procedure with low-cardinality outputs
  4. The output format : structured, with reasoningbefore the verdict

One Criterion per Judge

“Rate the response 1–10 for quality” is the most common judge prompt and the least useful. The judge silently weighs a dozen things differently on every call. Split it:

Criterion Question
Factual accuracy Does every factual claim match the tool outputs?
Groundedness Does it state any policy or promise not supported by the KB?
Resolution clarity Would the customer know exactly what happened and what happens next?
Tone Is it appropriate for the customer’s emotional state?

Four narrow judges are more reliable than one broad one, and when one fails you know what failed.

Binary or Low-Cardinality Scales

Judges are much more consistent on pass/fail or 3-point scales than on 1–10. The difference between a 6 and a 7 isn’t something a model (or a human) applies consistently. If you need gradation, use an analytic rubric: several binary checks that you sum.

Criterion: Factual accuracy

Answer each question YES or NO based only on the TOOL OUTPUTS and RESPONSE.
Q1. Does every amount mentioned in the response appear in the tool outputs?
Q2. Does every invoice or date mentioned match the tool outputs?
Q3. If the response says an action was taken, does a tool output confirm it succeeded?
Q4. Does the response avoid claiming facts about the account that no tool returned?

PASS only if all four are YES.

This is far more reproducible than “rate accuracy 1–5,” and the per-question answers double as a diagnostic.

Show Only the Evidence Needed

Give the judge the inputs it needs to decide the criterion, not the entire trace. More context means more distraction and more cost. For factual accuracy: the user message, the tool outputs, and the response. For tone: the user message and the response. The system prompt usually isn’t needed, and including it can bias the judge toward the agent’s own framing.

Reasoning Before Verdict

Ask for reasoning first, then the verdict, in structured output. Verdict-first outputs get rationalized after the fact.

Implementing a Judge Grader #

import json
from dataclasses import dataclass

JUDGE_TEMPLATE = """You are grading a customer-support agent's reply for ONE criterion.

## Criterion
{criterion}

## Rubric
{rubric}

## Evidence
<user_message>
{user_message}
</user_message>

<tool_outputs>
{tool_outputs}
</tool_outputs>

<agent_response>
{response}
</agent_response>

{case_notes}

Respond with JSON only:
{{"answers": {{"Q1": "YES|NO", ...}}, "reasoning": "<brief>", "verdict": "PASS|FAIL"}}
"""

@dataclass
class RubricJudge:
    name: str
    criterion: str
    rubric: str
    judge_llm: callable        # (prompt: str) -> str, temperature 0
    include_tools: bool = True

    def grade(self, case, run) -> Score:
        tool_outputs = "\n".join(
            f"{s.name}({json.dumps(s.input)}) -> {json.dumps(s.output)}"
            for s in run.trace.of_kind("tool")
        ) if self.include_tools else "(not shown)"
        notes = case.expected.get("rubric_notes", "")
        prompt = JUDGE_TEMPLATE.format(
            criterion=self.criterion, rubric=self.rubric,
            user_message=case.input, tool_outputs=tool_outputs,
            response=run.response,
            case_notes=f"## Case-specific notes\n{notes}" if notes else "",
        )
        raw = self.judge_llm(prompt)
        try:
            out = json.loads(extract_json(raw))
        except ValueError:
            return Score(self.name, 0.0, False, "judge returned unparseable output",
                         {"raw": raw})
        yes = sum(v == "YES" for v in out.get("answers", {}).values())
        total = max(len(out.get("answers", {})), 1)
        passed = out.get("verdict") == "PASS"
        return Score(self.name, yes / total, passed, out.get("reasoning", ""),
                     {"answers": out.get("answers")})

Note the rubric_notes hook from Lesson 4: case authors can add per-case guidance (“the customer explicitly asked not to be escalated”) without changing the judge.

Pointwise, Pairwise, and Reference-Guided #

Mode How it works Best for
Pointwise Judge one response against a rubric Absolute quality gates; monitoring
Pairwise Judge which of two responses is better Comparing two agent versions; subtle quality differences
Reference-guided Judge against an expert reference answer Factual tasks where an ideal answer exists

Pairwise is more sensitive than pointwise for small differences. Humans and models are both better at “which is better?” than “how good is this?” Use it when evaluating whether a prompt change improved tone or clarity. Report it as a win rate with ties allowed.

Reference-guided judging helps when a correct answer exists but can be phrased many ways. Tell the judge the reference is one acceptable answer, not the answer, or it will penalize valid alternatives.

Known Judge Biases #

These are well documented in the research literature and show up in practice. Test for each one.

Position bias. In pairwise mode, judges prefer whichever response appears first (or second, depending on the model). Mitigation: run every pair in both orders. Count a win only if it holds in both; otherwise it’s a tie.

def pairwise(judge, case, run_a, run_b) -> str:
    v1 = judge.compare(case, first=run_a, second=run_b)   # "first" | "second" | "tie"
    v2 = judge.compare(case, first=run_b, second=run_a)
    if v1 == "first" and v2 == "second": return "A"
    if v1 == "second" and v2 == "first": return "B"
    return "tie"

Verbosity bias. Judges favor longer responses, even when the extra length adds nothing. Mitigation: put concision in the rubric explicitly; check the correlation between judge score and response length on your data. If it’s strong and humans don’t share it, the judge is biased.

Self-preference bias. Models tend to rate outputs from their own family higher. Mitigation: where possible, use a judge from a different model family than the agent, or at least validate that the judge doesn’t systematically prefer one version for reasons humans don’t.

Sycophancy toward confident text. Assertive, well-formatted wrong answers get rated higher than hedged correct ones. Mitigation: give the judge the tool outputs so it can check facts, instead of asking it to assess plausibility.

Rubric drift across model versions. Upgrade the judge model, and your scores shift even though the agent didn’t change. Mitigation: pin the judge model version, and treat a judge upgrade like an agent change: recalibrate before adopting.

Calibration: Proving the Judge Works #

This is the step most teams skip, and it’s the one that makes the judge’s numbers mean anything.

Step 1: Collect Human Labels

Take 100–200 runs that span the range of quality: clear passes, clear fails, and borderline cases. Have domain experts label them using the same rubric the judge uses. Two labelers per item if you can afford it, so you can measure human-human agreement too.

Step 2: Measure Agreement

from sklearn.metrics import cohen_kappa_score, confusion_matrix

def calibration_report(human: list[bool], judge: list[bool]) -> dict:
    tn, fp, fn, tp = confusion_matrix(human, judge, labels=[False, True]).ravel()
    return {
        "agreement": (tp + tn) / len(human),
        "cohen_kappa": cohen_kappa_score(human, judge),
        "judge_false_pass": fp / (fp + tn) if fp + tn else 0.0,   # judge PASS, human FAIL
        "judge_false_fail": fn / (fn + tp) if fn + tp else 0.0,   # judge FAIL, human PASS
    }

Raw agreement is misleading when most items pass. If 90% of responses are good, a judge that always says PASS gets 90% agreement. Cohen’s kappa corrects for chance. Rough guide:

Kappa Interpretation
below 0.4 Don’t use this judge for decisions
0.4–0.6 Moderate; useful for trends, not gates
0.6–0.8 Substantial; usable for most purposes
above 0.8 Excellent; comparable to good human-human agreement

Also compare to human-human kappa. If two experts only agree at 0.65, a judge at 0.62 is about as good as you’ll get, and the rubric (not the judge) may need clarifying.

Step 3: Read the Disagreements

The confusion matrix tells you how often the judge is wrong. Reading the disagreements tells you why. Typical findings:

  • The rubric was ambiguous, and humans interpreted it differently from the judge. Fix the rubric.
  • The judge lacked evidence (e.g., couldn’t see a tool output). Fix the evidence.
  • The judge has a systematic bias (verbosity, leniency). Add explicit rubric language or few-shot examples of the boundary.

Iterate the rubric, rerun on the labeled set, and recompute kappa. Hold out a portion of the labels so you don’t overfit the rubric to them.

Step 4: Decide What the Judge Is Allowed to Do

Asymmetric errors matter. For a safety-relevant criterion, a judge with a 10% false-pass rate is dangerous; for tone, it’s fine. Write down, per judge:

  • Is it a gate orweighted ?
  • What kappa and false-pass rate did it achieve, on which label set, with which judge model version?
  • When must it be recalibrated? (On any judge-model change, rubric change, or quarterly.)

Cost and Practicality #

  • Use a smaller judge where it calibrates well. A small, fast model often matches a large one on narrow binary criteria. Calibration tells you whether it does.
  • Cache judge verdicts keyed on (criterion version, judge model, evidence hash). Rerunning a report shouldn’t re-pay for identical judgments.
  • Run judges only on runs that passed the gates. If the agent refunded the wrong invoice, there’s no point grading its tone.
  • Sample for expensive judges. For monitoring, judging a random 10% of traffic is often enough to detect trends (Lesson 11).

Key Takeaways #

  • Use judges only for what code can’t check: accuracy against evidence, groundedness, clarity, tone
  • One criterion per judge; binary or analytic rubrics; reasoning before verdict; only the evidence needed
  • Pairwise is more sensitive for comparisons; always run both orders to cancel position bias
  • Test for position, verbosity, and self-preference bias; pin the judge model version
  • Calibrate against human labels with Cohen’s kappa, read the disagreements, and document what each judge is trusted to decide

Exercise #

  1. Write two rubric judges for your agent: factual accuracy (with tool outputs as evidence) and resolution clarity
  2. Label 60 runs yourself (or with a colleague) using the same rubrics
  3. Compute agreement, kappa, false-pass, and false-fail rates for each judge
  4. Read every disagreement, revise the rubric once, and recompute on 20 held-out labels
  5. Record each judge’s calibration results in a JUDGES.md file next to your graders

If either judge’s kappa stays below 0.5, don’t use it as a gate. Make it a weighted signal and keep improving the rubric.

── more in #ai-agents 4 stories · sorted by recency
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/llm-as-judge-done-ri…] indexed:0 read:10min 2026-11-18 · —