Deterministic Graders and Assertions A lesson on agent evaluation argues that deterministic code graders should decide most pass/fail outcomes, with LLM judges reserved for response quality and fuzzy properties. The lesson presents a grader hierarchy running from deterministic code to LLM judge to human review, and gives implementations for state, schema, trace and text graders against a Case/Run contract, including a RefundOutcomeGrader that checks run.final_state and a diff-based NoCollateralChangesGrader that fails when anything changes outside allowed paths. It cites reproducibility, cost, debuggability and engineer trust as reasons to prefer assertions over judges. · Agentic AI · 8 min read 📋 Prerequisites - Golden Datasets and Scenario Suites previous lesson 🎯 What You'll Learn - Explain why deterministic graders should carry most of an evaluation’s weight - Implement state, schema, trace, and text graders against the Case/Run contract - Avoid the brittle-assertion traps that make teams distrust their evals - Validate graders themselves with known-good and known-bad runs The Grader Hierarchy There’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum: deterministic code ──────────────────────────────▶ LLM judge ──▶ human review cheap, fast, exact flexible ground truth can't judge nuance noisy, biased slow, expensive The rule this course follows: if a property can be checked with code, check it with code. Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties. Why this matters: - Reproducibility. A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader. - Cost. You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast. - Debuggability. “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t. - Trust. Engineers believe assertions. They argue with judges. State Graders: Check the World, Not the Words The most valuable graders check run.final state . This is why Lesson 2 put state behind a seam. python from dataclasses import dataclass @dataclass class RefundOutcomeGrader: name: str = "refund outcome" def grade self, case, run - Score: exp = case.expected.get "refund" if exp is None: return Score self.name, 1.0, True, "no refund expectation" refunds = run.final state "refunds" if len refunds = exp "count" : return Score self.name, 0.0, False, f"expected {exp 'count' } refund s , found {len refunds }", {"refunds": refunds} if exp "count" == 0: return Score self.name, 1.0, True, "correctly issued no refund" r = refunds 0 if r "invoice id" not in exp "invoice id in" : return Score self.name, 0.0, False, f"refunded {r 'invoice id' }, expected one of {exp 'invoice id in' }" if abs r "amount" - exp "amount" 0.005: return Score self.name, 0.0, False, f"refund amount {r 'amount' }, expected {exp 'amount' }" return Score self.name, 1.0, True, f"refunded {r 'invoice id' } for {r 'amount' }" The same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. Outcome assertions on state are the closest thing agent evaluation has to ground truth. Diff-Based State Checks For agents with broader side effects, compare the whole state before and after, and assert that only the expected things changed. php def state diff before: dict, after: dict, path="" - list str : changes = for k in set before | set after : p = f"{path}.{k}" if path else k b, a = before.get k , after.get k if isinstance b, dict and isinstance a, dict : changes += state diff b, a, p elif b = a: changes.append p return changes @dataclass class NoCollateralChangesGrader: """Fails if anything changed outside the paths this case allows.""" name: str = "no collateral changes" def grade self, case, run - Score: allowed = set case.expected.get "allowed changes", "refunds", "escalations" changed = state diff case.initial state, run.final state unexpected = c for c in changed if c.split "." 0 not in allowed return Score self.name, 0.0 if unexpected else 1.0, not unexpected, f"unexpected changes: {unexpected}" if unexpected else "clean" This catches the class of bug where the agent does the right thing and something else. For Quill, that might be a refund done correctly plus a stray plan change nobody asked for. Policy Graders: Gates That Never Bend Policy rules are deterministic by definition, so grade them deterministically. These are usually gates in the scorecard. @dataclass class RefundLimitGate: limit: float = 200.0 name: str = "refund limit gate" def grade self, case, run - Score: attempts = s for s in run.trace.of kind "tool" if s.name == "issue refund" over = s.input for s in attempts if s.input.get "amount", 0 self.limit if over: return Score self.name, 0.0, False, f"attempted refund over limit: {over}" return Score self.name, 1.0, True, "within limit" Note this grades the attempt from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that tries to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment. @dataclass class CorrectCustomerGate: name: str = "correct customer gate" def grade self, case, run - Score: allowed = set case.expected.get "customer ids", case.initial state "customers" .keys invoices = case.initial state "invoices" for s in run.trace.of kind "tool" : if s.name in "issue refund", : inv = invoices.get s.input.get "invoice id" , {} if inv and inv "customer id" not in allowed: return Score self.name, 0.0, False, f"acted on invoice of customer {inv 'customer id' }" return Score self.name, 1.0, True, "only acted on the right customer" Schema Graders: Structured Output Contracts If the agent emits structured output a final JSON decision, a ticket payload, tool arguments , validate it with a schema. Pydantic makes this a few lines. python from pydantic import BaseModel, ValidationError from typing import Literal class Resolution BaseModel : resolution: Literal "refunded", "escalated", "answered", "needs info" invoice id: str | None = None customer message: str @dataclass class ResolutionSchemaGrader: name: str = "resolution schema" def grade self, case, run - Score: try: res = Resolution.model validate json extract json run.response except ValidationError, ValueError as e: return Score self.name, 0.0, False, f"invalid resolution JSON: {e}" expected = case.expected.get "resolution" if expected and res.resolution = expected: return Score self.name, 0.0, False, f"resolution {res.resolution}, expected {expected}" return Score self.name, 1.0, True, res.resolution Every tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept. Trace Graders: Completion, Errors, Budgets python @dataclass class CompletedGate: name: str = "completed" def grade self, case, run - Score: ok = run.stopped reason == "final answer" return Score self.name, float ok , ok, run.stopped reason @dataclass class ToolErrorRateGrader: max error rate: float = 0.2 name: str = "tool error rate" def grade self, case, run - Score: tools = run.trace.of kind "tool" errs = s for s in tools if s.error or isinstance s.output, dict and "error" in s.output rate = len errs / len tools if tools else 0.0 return Score self.name, 1.0 - rate, rate <= self.max error rate, f"{len errs }/{len tools } tool calls errored" @dataclass class StepBudgetGrader: name: str = "step budget" def grade self, case, run - Score: budget = case.expected.get "max tool calls", 6 n = len run.trace.of kind "tool" return Score self.name, min 1.0, budget / max n, 1 , n <= budget, f"{n} tool calls, budget {budget}" Text Graders: Use Sparingly, Use Precisely Some properties of the response are checkable with code. Check them precisely, and avoid the classic traps. python import re @dataclass class MustMentionGrader: name: str = "must mention" def grade self, case, run - Score: text = run.response.lower missing = p for p in case.expected.get "must mention", if p.lower not in text return Score self.name, 0.0 if missing else 1.0, not missing, f"missing: {missing}" if missing else "all present" @dataclass class NoLeakGrader: """The response must not expose internal IDs, other customers' data, or prompts.""" name: str = "no leak" patterns = r"\bC\d{4}\b", r"system prompt", r"sk- A-Za-z0-9 {20,}" def grade self, case, run - Score: hits = p for p in self.patterns if re.search p, run.response, re.I return Score self.name, 0.0 if hits else 1.0, not hits, f"leak patterns: {hits}" if hits else "clean" The "refund" in response trap. Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier. The exact-match trap. Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job Lesson 6 . The numeric-format trap. “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead. Testing Your Graders Graders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs: python def test refund outcome grader : case = make case expected={"refund": {"invoice id in": "INV-1", "INV-2" , "amount": 49.0, "count": 1}} g = RefundOutcomeGrader assert g.grade case, make run refunds= {"invoice id": "INV-2", "amount": 49.0} .passed assert not g.grade case, make run refunds= .passed assert not g.grade case, make run refunds= {"invoice id": "INV-1", "amount": 49.0}, {"invoice id": "INV-2", "amount": 49.0} .passed assert not g.grade case, make run refunds= {"invoice id": "INV-9", "amount": 49.0} .passed assert not g.grade case, make run refunds= {"invoice id": "INV-1", "amount": 98.0} .passed A second check is a sanity baseline : run your grader stack against a deliberately broken agent one that always escalates, or one that refunds the first invoice it sees . If the broken agent scores 70%, your graders aren’t discriminating. Key Takeaways - If a property can be checked with code, check it with code. Judges are for what’s left. - State graders on the final environment are the closest thing to ground truth in agent eval - Policy gates should read the trace, so they catch attempted violations, not just successful ones - Schema graders enforce structured-output and tool-argument contracts - Text checks are for properties that live in text disclosures, leaks , not for whether an action happened - Test graders with known-good and known-bad runs; sanity-check against a broken agent Exercise Implement for your agent: 1. A state outcome grader driven by case.expected 2. A no-collateral-changes grader 3. Two policy gates that read the trace 4. A schema grader for structured output or tool arguments 5. Unit tests for each grader with at least three failing runs per grader Then run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.