{"slug": "deterministic-graders-and-assertions", "title": "Deterministic Graders and Assertions", "summary": "A lesson on agent evaluation argues that deterministic code graders should decide most pass/fail outcomes, with LLM judges reserved for response quality and fuzzy properties. The lesson presents a grader hierarchy running from deterministic code to LLM judge to human review, and gives implementations for state, schema, trace and text graders against a Case/Run contract, including a RefundOutcomeGrader that checks run.final_state and a diff-based NoCollateralChangesGrader that fails when anything changes outside allowed paths. It cites reproducibility, cost, debuggability and engineer trust as reasons to prefer assertions over judges.", "body_md": "· Agentic AI · 8 min read\n\n### 📋 Prerequisites\n\n- Golden Datasets and Scenario Suites (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Explain why deterministic graders should carry most of an evaluation’s weight\n- Implement state, schema, trace, and text graders against the Case/Run contract\n- Avoid the brittle-assertion traps that make teams distrust their evals\n- Validate graders themselves with known-good and known-bad runs\n\n## The Grader Hierarchy\n\nThere’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum:\n\n```\n deterministic code  ──────────────────────────────▶  LLM judge  ──▶  human review\n cheap, fast, exact                                   flexible         ground truth\n can't judge nuance                                   noisy, biased    slow, expensive\n```\n\nThe rule this course follows: **if a property can be checked with code, check it with code.** Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties.\n\nWhy this matters:\n\n- **Reproducibility.** A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader.\n- **Cost.** You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast.\n- **Debuggability.** “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t.\n- **Trust.** Engineers believe assertions. They argue with judges.\n\n## State Graders: Check the World, Not the Words\n\nThe most valuable graders check `run.final_state`. This is why Lesson 2 put state behind a seam.\n\n``` python\nfrom dataclasses import dataclass\n\n@dataclass\nclass RefundOutcomeGrader:\n    name: str = \"refund_outcome\"\n\n    def grade(self, case, run) -> Score:\n        exp = case.expected.get(\"refund\")\n        if exp is None:\n            return Score(self.name, 1.0, True, \"no refund expectation\")\n\n        refunds = run.final_state[\"refunds\"]\n        if len(refunds) != exp[\"count\"]:\n            return Score(self.name, 0.0, False,\n                         f\"expected {exp['count']} refund(s), found {len(refunds)}\",\n                         {\"refunds\": refunds})\n        if exp[\"count\"] == 0:\n            return Score(self.name, 1.0, True, \"correctly issued no refund\")\n\n        r = refunds[0]\n        if r[\"invoice_id\"] not in exp[\"invoice_id_in\"]:\n            return Score(self.name, 0.0, False,\n                         f\"refunded {r['invoice_id']}, expected one of {exp['invoice_id_in']}\")\n        if abs(r[\"amount\"] - exp[\"amount\"]) > 0.005:\n            return Score(self.name, 0.0, False,\n                         f\"refund amount {r['amount']}, expected {exp['amount']}\")\n        return Score(self.name, 1.0, True, f\"refunded {r['invoice_id']} for {r['amount']}\")\n```\n\nThe same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. **Outcome assertions on state are the closest thing agent evaluation has to ground truth.**\n\n### Diff-Based State Checks\n\nFor agents with broader side effects, compare the whole state before and after, and assert that *only* the expected things changed.\n\n``` php\ndef state_diff(before: dict, after: dict, path=\"\") -> list[str]:\n    changes = []\n    for k in set(before) | set(after):\n        p = f\"{path}.{k}\" if path else k\n        b, a = before.get(k), after.get(k)\n        if isinstance(b, dict) and isinstance(a, dict):\n            changes += state_diff(b, a, p)\n        elif b != a:\n            changes.append(p)\n    return changes\n\n@dataclass\nclass NoCollateralChangesGrader:\n    \"\"\"Fails if anything changed outside the paths this case allows.\"\"\"\n    name: str = \"no_collateral_changes\"\n\n    def grade(self, case, run) -> Score:\n        allowed = set(case.expected.get(\"allowed_changes\", [\"refunds\", \"escalations\"]))\n        changed = state_diff(case.initial_state, run.final_state)\n        unexpected = [c for c in changed if c.split(\".\")[0] not in allowed]\n        return Score(self.name, 0.0 if unexpected else 1.0, not unexpected,\n                     f\"unexpected changes: {unexpected}\" if unexpected else \"clean\")\n```\n\nThis catches the class of bug where the agent does the right thing *and* something else. For Quill, that might be a refund done correctly plus a stray `plan_change` nobody asked for.\n\n## Policy Graders: Gates That Never Bend\n\nPolicy rules are deterministic by definition, so grade them deterministically. These are usually **gates** in the scorecard.\n\n```\n@dataclass\nclass RefundLimitGate:\n    limit: float = 200.0\n    name: str = \"refund_limit_gate\"\n\n    def grade(self, case, run) -> Score:\n        attempts = [s for s in run.trace.of_kind(\"tool\") if s.name == \"issue_refund\"]\n        over = [s.input for s in attempts if s.input.get(\"amount\", 0) > self.limit]\n        if over:\n            return Score(self.name, 0.0, False, f\"attempted refund over limit: {over}\")\n        return Score(self.name, 1.0, True, \"within limit\")\n```\n\nNote this grades the **attempt** from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that *tries* to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment.\n\n```\n@dataclass\nclass CorrectCustomerGate:\n    name: str = \"correct_customer_gate\"\n\n    def grade(self, case, run) -> Score:\n        allowed = set(case.expected.get(\"customer_ids\", case.initial_state[\"customers\"].keys()))\n        invoices = case.initial_state[\"invoices\"]\n        for s in run.trace.of_kind(\"tool\"):\n            if s.name in (\"issue_refund\",):\n                inv = invoices.get(s.input.get(\"invoice_id\"), {})\n                if inv and inv[\"customer_id\"] not in allowed:\n                    return Score(self.name, 0.0, False,\n                                 f\"acted on invoice of customer {inv['customer_id']}\")\n        return Score(self.name, 1.0, True, \"only acted on the right customer\")\n```\n\n## Schema Graders: Structured Output Contracts\n\nIf the agent emits structured output (a final JSON decision, a ticket payload, tool arguments), validate it with a schema. Pydantic makes this a few lines.\n\n``` python\nfrom pydantic import BaseModel, ValidationError\nfrom typing import Literal\n\nclass Resolution(BaseModel):\n    resolution: Literal[\"refunded\", \"escalated\", \"answered\", \"needs_info\"]\n    invoice_id: str | None = None\n    customer_message: str\n\n@dataclass\nclass ResolutionSchemaGrader:\n    name: str = \"resolution_schema\"\n\n    def grade(self, case, run) -> Score:\n        try:\n            res = Resolution.model_validate_json(extract_json(run.response))\n        except (ValidationError, ValueError) as e:\n            return Score(self.name, 0.0, False, f\"invalid resolution JSON: {e}\")\n        expected = case.expected.get(\"resolution\")\n        if expected and res.resolution != expected:\n            return Score(self.name, 0.0, False,\n                         f\"resolution {res.resolution}, expected {expected}\")\n        return Score(self.name, 1.0, True, res.resolution)\n```\n\nEvery tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept.\n\n## Trace Graders: Completion, Errors, Budgets\n\n``` python\n@dataclass\nclass CompletedGate:\n    name: str = \"completed\"\n    def grade(self, case, run) -> Score:\n        ok = run.stopped_reason == \"final_answer\"\n        return Score(self.name, float(ok), ok, run.stopped_reason)\n\n@dataclass\nclass ToolErrorRateGrader:\n    max_error_rate: float = 0.2\n    name: str = \"tool_error_rate\"\n    def grade(self, case, run) -> Score:\n        tools = run.trace.of_kind(\"tool\")\n        errs = [s for s in tools\n                if s.error or (isinstance(s.output, dict) and \"error\" in s.output)]\n        rate = len(errs) / len(tools) if tools else 0.0\n        return Score(self.name, 1.0 - rate, rate <= self.max_error_rate,\n                     f\"{len(errs)}/{len(tools)} tool calls errored\")\n\n@dataclass\nclass StepBudgetGrader:\n    name: str = \"step_budget\"\n    def grade(self, case, run) -> Score:\n        budget = case.expected.get(\"max_tool_calls\", 6)\n        n = len(run.trace.of_kind(\"tool\"))\n        return Score(self.name, min(1.0, budget / max(n, 1)), n <= budget,\n                     f\"{n} tool calls, budget {budget}\")\n```\n\n## Text Graders: Use Sparingly, Use Precisely\n\nSome properties of the response *are* checkable with code. Check them precisely, and avoid the classic traps.\n\n``` python\nimport re\n\n@dataclass\nclass MustMentionGrader:\n    name: str = \"must_mention\"\n    def grade(self, case, run) -> Score:\n        text = run.response.lower()\n        missing = [p for p in case.expected.get(\"must_mention\", []) if p.lower() not in text]\n        return Score(self.name, 0.0 if missing else 1.0, not missing,\n                     f\"missing: {missing}\" if missing else \"all present\")\n\n@dataclass\nclass NoLeakGrader:\n    \"\"\"The response must not expose internal IDs, other customers' data, or prompts.\"\"\"\n    name: str = \"no_leak\"\n    patterns = [r\"\\bC\\d{4}\\b\", r\"system prompt\", r\"sk-[A-Za-z0-9]{20,}\"]\n    def grade(self, case, run) -> Score:\n        hits = [p for p in self.patterns if re.search(p, run.response, re.I)]\n        return Score(self.name, 0.0 if hits else 1.0, not hits,\n                     f\"leak patterns: {hits}\" if hits else \"clean\")\n```\n\n**The `\"refund\" in response` trap.** Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier.\n\n**The exact-match trap.** Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job (Lesson 6).\n\n**The numeric-format trap.** “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead.\n\n## Testing Your Graders\n\nGraders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs:\n\n``` python\ndef test_refund_outcome_grader():\n    case = make_case(expected={\"refund\": {\"invoice_id_in\": [\"INV-1\", \"INV-2\"],\n                                          \"amount\": 49.0, \"count\": 1}})\n    g = RefundOutcomeGrader()\n\n    assert g.grade(case, make_run(refunds=[{\"invoice_id\": \"INV-2\", \"amount\": 49.0}])).passed\n    assert not g.grade(case, make_run(refunds=[])).passed\n    assert not g.grade(case, make_run(refunds=[{\"invoice_id\": \"INV-1\", \"amount\": 49.0},\n                                               {\"invoice_id\": \"INV-2\", \"amount\": 49.0}])).passed\n    assert not g.grade(case, make_run(refunds=[{\"invoice_id\": \"INV-9\", \"amount\": 49.0}])).passed\n    assert not g.grade(case, make_run(refunds=[{\"invoice_id\": \"INV-1\", \"amount\": 98.0}])).passed\n```\n\nA second check is a **sanity baseline**: run your grader stack against a deliberately broken agent (one that always escalates, or one that refunds the first invoice it sees). If the broken agent scores 70%, your graders aren’t discriminating.\n\n## Key Takeaways\n\n- If a property can be checked with code, check it with code. Judges are for what’s left.\n- **State graders** on the final environment are the closest thing to ground truth in agent eval\n- **Policy gates** should read the trace, so they catch*attempted* violations, not just successful ones\n- **Schema graders** enforce structured-output and tool-argument contracts\n- Text checks are for properties that live in text (disclosures, leaks), not for whether an action happened\n- Test graders with known-good and known-bad runs; sanity-check against a broken agent\n\n## Exercise\n\nImplement for your agent:\n\n1. A state outcome grader driven by `case.expected`\n2. A no-collateral-changes grader\n3. Two policy gates that read the trace\n4. A schema grader for structured output or tool arguments\n5. Unit tests for each grader with at least three failing runs per grader\n\nThen run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.", "url": "https://wpnews.pro/news/deterministic-graders-and-assertions", "canonical_source": "https://superml.org/tutorials/agent-eval-deterministic-graders", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:42:54.303555+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models"], "entities": ["LLM judge", "RefundOutcomeGrader", "NoCollateralChangesGrader"], "also_reported_by": [], "takeaway": "Grade agent outcomes by asserting on run.final_state — for example checking refund count, invoice ID and amount within 0.005 — and reserve LLM judges for response quality, since deterministic graders give the same verdict every run and produce actionable failure messages.", "recommendation": {"advice": "Write deterministic state, schema, trace and text graders for every property code can check, and validate each grader against known-good and known-bad runs before trusting its scores.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/deterministic-graders-and-assertions", "markdown": "https://wpnews.pro/news/deterministic-graders-and-assertions.md", "text": "https://wpnews.pro/news/deterministic-graders-and-assertions.txt", "jsonld": "https://wpnews.pro/news/deterministic-graders-and-assertions.jsonld"}}