· Agentic AI · 9 min read
📋 Prerequisites
- Deterministic Graders and Assertions (previous lesson)
🎯 What You'll Learn
- Decide when an LLM judge is the right grader and when it is not
- Write analytic rubrics with binary or low-cardinality criteria that judges apply consistently
- Choose between pointwise, pairwise, and reference-guided judging
- Measure and mitigate position, verbosity, and self-preference bias
- Calibrate a judge against human labels using agreement metrics
What Judges Are For #
After Lesson 5, your deterministic graders handle outcomes, policies, schemas, and budgets. What’s left is the stuff code can’t check:
- Is the response accurate relative to what the tools returned?
- Is it grounded , or does it invent policy?
- Is it clear and appropriate in tone for an upset customer?
- Did the agent ask for clarification when the request was genuinely ambiguous?
- Is the reasoning in a long trajectory sound , even when the outcome was right?
These are judgment calls. An LLM judge can make them at scale, if you design the judge carefully and prove that it agrees with humans. An unvalidated judge is a random number generator with excellent grammar.
Anatomy of a Good Judge #
A judge has four parts:
- The criterion : one specific property, not “overall quality”
- The evidence : exactly what the judge needs to see (the user message, tool outputs, the response), and nothing more
- The rubric : an explicit decision procedure with low-cardinality outputs
- The output format : structured, with reasoningbefore the verdict
One Criterion per Judge
“Rate the response 1–10 for quality” is the most common judge prompt and the least useful. The judge silently weighs a dozen things differently on every call. Split it:
| Criterion | Question |
|---|---|
| Factual accuracy | Does every factual claim match the tool outputs? |
| Groundedness | Does it state any policy or promise not supported by the KB? |
| Resolution clarity | Would the customer know exactly what happened and what happens next? |
| Tone | Is it appropriate for the customer’s emotional state? |
Four narrow judges are more reliable than one broad one, and when one fails you know what failed.
Binary or Low-Cardinality Scales
Judges are much more consistent on pass/fail or 3-point scales than on 1–10. The difference between a 6 and a 7 isn’t something a model (or a human) applies consistently. If you need gradation, use an analytic rubric: several binary checks that you sum.
Criterion: Factual accuracy
Answer each question YES or NO based only on the TOOL OUTPUTS and RESPONSE.
Q1. Does every amount mentioned in the response appear in the tool outputs?
Q2. Does every invoice or date mentioned match the tool outputs?
Q3. If the response says an action was taken, does a tool output confirm it succeeded?
Q4. Does the response avoid claiming facts about the account that no tool returned?
PASS only if all four are YES.
This is far more reproducible than “rate accuracy 1–5,” and the per-question answers double as a diagnostic.
Show Only the Evidence Needed
Give the judge the inputs it needs to decide the criterion, not the entire trace. More context means more distraction and more cost. For factual accuracy: the user message, the tool outputs, and the response. For tone: the user message and the response. The system prompt usually isn’t needed, and including it can bias the judge toward the agent’s own framing.
Reasoning Before Verdict
Ask for reasoning first, then the verdict, in structured output. Verdict-first outputs get rationalized after the fact.
Implementing a Judge Grader #
import json
from dataclasses import dataclass
JUDGE_TEMPLATE = """You are grading a customer-support agent's reply for ONE criterion.
## Criterion
{criterion}
## Rubric
{rubric}
## Evidence
<user_message>
{user_message}
</user_message>
<tool_outputs>
{tool_outputs}
</tool_outputs>
<agent_response>
{response}
</agent_response>
{case_notes}
Respond with JSON only:
{{"answers": {{"Q1": "YES|NO", ...}}, "reasoning": "<brief>", "verdict": "PASS|FAIL"}}
"""
@dataclass
class RubricJudge:
name: str
criterion: str
rubric: str
judge_llm: callable # (prompt: str) -> str, temperature 0
include_tools: bool = True
def grade(self, case, run) -> Score:
tool_outputs = "\n".join(
f"{s.name}({json.dumps(s.input)}) -> {json.dumps(s.output)}"
for s in run.trace.of_kind("tool")
) if self.include_tools else "(not shown)"
notes = case.expected.get("rubric_notes", "")
prompt = JUDGE_TEMPLATE.format(
criterion=self.criterion, rubric=self.rubric,
user_message=case.input, tool_outputs=tool_outputs,
response=run.response,
case_notes=f"## Case-specific notes\n{notes}" if notes else "",
)
raw = self.judge_llm(prompt)
try:
out = json.loads(extract_json(raw))
except ValueError:
return Score(self.name, 0.0, False, "judge returned unparseable output",
{"raw": raw})
yes = sum(v == "YES" for v in out.get("answers", {}).values())
total = max(len(out.get("answers", {})), 1)
passed = out.get("verdict") == "PASS"
return Score(self.name, yes / total, passed, out.get("reasoning", ""),
{"answers": out.get("answers")})
Note the rubric_notes hook from Lesson 4: case authors can add per-case guidance (“the customer explicitly asked not to be escalated”) without changing the judge.
Pointwise, Pairwise, and Reference-Guided #
| Mode | How it works | Best for |
|---|---|---|
| Pointwise | Judge one response against a rubric | Absolute quality gates; monitoring |
| Pairwise | Judge which of two responses is better | Comparing two agent versions; subtle quality differences |
| Reference-guided | Judge against an expert reference answer | Factual tasks where an ideal answer exists |
Pairwise is more sensitive than pointwise for small differences. Humans and models are both better at “which is better?” than “how good is this?” Use it when evaluating whether a prompt change improved tone or clarity. Report it as a win rate with ties allowed.
Reference-guided judging helps when a correct answer exists but can be phrased many ways. Tell the judge the reference is one acceptable answer, not the answer, or it will penalize valid alternatives.
Known Judge Biases #
These are well documented in the research literature and show up in practice. Test for each one.
Position bias. In pairwise mode, judges prefer whichever response appears first (or second, depending on the model). Mitigation: run every pair in both orders. Count a win only if it holds in both; otherwise it’s a tie.
def pairwise(judge, case, run_a, run_b) -> str:
v1 = judge.compare(case, first=run_a, second=run_b) # "first" | "second" | "tie"
v2 = judge.compare(case, first=run_b, second=run_a)
if v1 == "first" and v2 == "second": return "A"
if v1 == "second" and v2 == "first": return "B"
return "tie"
Verbosity bias. Judges favor longer responses, even when the extra length adds nothing. Mitigation: put concision in the rubric explicitly; check the correlation between judge score and response length on your data. If it’s strong and humans don’t share it, the judge is biased.
Self-preference bias. Models tend to rate outputs from their own family higher. Mitigation: where possible, use a judge from a different model family than the agent, or at least validate that the judge doesn’t systematically prefer one version for reasons humans don’t.
Sycophancy toward confident text. Assertive, well-formatted wrong answers get rated higher than hedged correct ones. Mitigation: give the judge the tool outputs so it can check facts, instead of asking it to assess plausibility.
Rubric drift across model versions. Upgrade the judge model, and your scores shift even though the agent didn’t change. Mitigation: pin the judge model version, and treat a judge upgrade like an agent change: recalibrate before adopting.
Calibration: Proving the Judge Works #
This is the step most teams skip, and it’s the one that makes the judge’s numbers mean anything.
Step 1: Collect Human Labels
Take 100–200 runs that span the range of quality: clear passes, clear fails, and borderline cases. Have domain experts label them using the same rubric the judge uses. Two labelers per item if you can afford it, so you can measure human-human agreement too.
Step 2: Measure Agreement
from sklearn.metrics import cohen_kappa_score, confusion_matrix
def calibration_report(human: list[bool], judge: list[bool]) -> dict:
tn, fp, fn, tp = confusion_matrix(human, judge, labels=[False, True]).ravel()
return {
"agreement": (tp + tn) / len(human),
"cohen_kappa": cohen_kappa_score(human, judge),
"judge_false_pass": fp / (fp + tn) if fp + tn else 0.0, # judge PASS, human FAIL
"judge_false_fail": fn / (fn + tp) if fn + tp else 0.0, # judge FAIL, human PASS
}
Raw agreement is misleading when most items pass. If 90% of responses are good, a judge that always says PASS gets 90% agreement. Cohen’s kappa corrects for chance. Rough guide:
| Kappa | Interpretation |
|---|---|
| below 0.4 | Don’t use this judge for decisions |
| 0.4–0.6 | Moderate; useful for trends, not gates |
| 0.6–0.8 | Substantial; usable for most purposes |
| above 0.8 | Excellent; comparable to good human-human agreement |
Also compare to human-human kappa. If two experts only agree at 0.65, a judge at 0.62 is about as good as you’ll get, and the rubric (not the judge) may need clarifying.
Step 3: Read the Disagreements
The confusion matrix tells you how often the judge is wrong. Reading the disagreements tells you why. Typical findings:
- The rubric was ambiguous, and humans interpreted it differently from the judge. Fix the rubric.
- The judge lacked evidence (e.g., couldn’t see a tool output). Fix the evidence.
- The judge has a systematic bias (verbosity, leniency). Add explicit rubric language or few-shot examples of the boundary.
Iterate the rubric, rerun on the labeled set, and recompute kappa. Hold out a portion of the labels so you don’t overfit the rubric to them.
Step 4: Decide What the Judge Is Allowed to Do
Asymmetric errors matter. For a safety-relevant criterion, a judge with a 10% false-pass rate is dangerous; for tone, it’s fine. Write down, per judge:
- Is it a gate orweighted ?
- What kappa and false-pass rate did it achieve, on which label set, with which judge model version?
- When must it be recalibrated? (On any judge-model change, rubric change, or quarterly.)
Cost and Practicality #
- Use a smaller judge where it calibrates well. A small, fast model often matches a large one on narrow binary criteria. Calibration tells you whether it does.
- Cache judge verdicts keyed on (criterion version, judge model, evidence hash). Rerunning a report shouldn’t re-pay for identical judgments.
- Run judges only on runs that passed the gates. If the agent refunded the wrong invoice, there’s no point grading its tone.
- Sample for expensive judges. For monitoring, judging a random 10% of traffic is often enough to detect trends (Lesson 11).
Key Takeaways #
- Use judges only for what code can’t check: accuracy against evidence, groundedness, clarity, tone
- One criterion per judge; binary or analytic rubrics; reasoning before verdict; only the evidence needed
- Pairwise is more sensitive for comparisons; always run both orders to cancel position bias
- Test for position, verbosity, and self-preference bias; pin the judge model version
- Calibrate against human labels with Cohen’s kappa, read the disagreements, and document what each judge is trusted to decide
Exercise #
- Write two rubric judges for your agent: factual accuracy (with tool outputs as evidence) and resolution clarity
- Label 60 runs yourself (or with a colleague) using the same rubrics
- Compute agreement, kappa, false-pass, and false-fail rates for each judge
- Read every disagreement, revise the rubric once, and recompute on 20 held-out labels
- Record each judge’s calibration results in a
JUDGES.mdfile next to your graders
If either judge’s kappa stays below 0.5, don’t use it as a gate. Make it a weighted signal and keep improving the rubric.