# Deterministic Graders and Assertions

> Source: <https://superml.org/tutorials/agent-eval-deterministic-graders>
> Published: 2026-11-18 00:00:00+00:00

**Recommendation:** Write deterministic state, schema, trace and text graders for every property code can check, and validate each grader against known-good and known-bad runs before trusting its scores.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

· Agentic AI · 8 min read

### 📋 Prerequisites

- Golden Datasets and Scenario Suites (previous lesson)

### 🎯 What You'll Learn

- Explain why deterministic graders should carry most of an evaluation’s weight
- Implement state, schema, trace, and text graders against the Case/Run contract
- Avoid the brittle-assertion traps that make teams distrust their evals
- Validate graders themselves with known-good and known-bad runs

## The Grader Hierarchy

There’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum:

```
 deterministic code  ──────────────────────────────▶  LLM judge  ──▶  human review
 cheap, fast, exact                                   flexible         ground truth
 can't judge nuance                                   noisy, biased    slow, expensive
```

The rule this course follows: **if a property can be checked with code, check it with code.** Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties.

Why this matters:

- **Reproducibility.** A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader.
- **Cost.** You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast.
- **Debuggability.** “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t.
- **Trust.** Engineers believe assertions. They argue with judges.

## State Graders: Check the World, Not the Words

The most valuable graders check `run.final_state`. This is why Lesson 2 put state behind a seam.

``` python
from dataclasses import dataclass

@dataclass
class RefundOutcomeGrader:
    name: str = "refund_outcome"

    def grade(self, case, run) -> Score:
        exp = case.expected.get("refund")
        if exp is None:
            return Score(self.name, 1.0, True, "no refund expectation")

        refunds = run.final_state["refunds"]
        if len(refunds) != exp["count"]:
            return Score(self.name, 0.0, False,
                         f"expected {exp['count']} refund(s), found {len(refunds)}",
                         {"refunds": refunds})
        if exp["count"] == 0:
            return Score(self.name, 1.0, True, "correctly issued no refund")

        r = refunds[0]
        if r["invoice_id"] not in exp["invoice_id_in"]:
            return Score(self.name, 0.0, False,
                         f"refunded {r['invoice_id']}, expected one of {exp['invoice_id_in']}")
        if abs(r["amount"] - exp["amount"]) > 0.005:
            return Score(self.name, 0.0, False,
                         f"refund amount {r['amount']}, expected {exp['amount']}")
        return Score(self.name, 1.0, True, f"refunded {r['invoice_id']} for {r['amount']}")
```

The same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. **Outcome assertions on state are the closest thing agent evaluation has to ground truth.**

### Diff-Based State Checks

For agents with broader side effects, compare the whole state before and after, and assert that *only* the expected things changed.

``` php
def state_diff(before: dict, after: dict, path="") -> list[str]:
    changes = []
    for k in set(before) | set(after):
        p = f"{path}.{k}" if path else k
        b, a = before.get(k), after.get(k)
        if isinstance(b, dict) and isinstance(a, dict):
            changes += state_diff(b, a, p)
        elif b != a:
            changes.append(p)
    return changes

@dataclass
class NoCollateralChangesGrader:
    """Fails if anything changed outside the paths this case allows."""
    name: str = "no_collateral_changes"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("allowed_changes", ["refunds", "escalations"]))
        changed = state_diff(case.initial_state, run.final_state)
        unexpected = [c for c in changed if c.split(".")[0] not in allowed]
        return Score(self.name, 0.0 if unexpected else 1.0, not unexpected,
                     f"unexpected changes: {unexpected}" if unexpected else "clean")
```

This catches the class of bug where the agent does the right thing *and* something else. For Quill, that might be a refund done correctly plus a stray `plan_change` nobody asked for.

## Policy Graders: Gates That Never Bend

Policy rules are deterministic by definition, so grade them deterministically. These are usually **gates** in the scorecard.

```
@dataclass
class RefundLimitGate:
    limit: float = 200.0
    name: str = "refund_limit_gate"

    def grade(self, case, run) -> Score:
        attempts = [s for s in run.trace.of_kind("tool") if s.name == "issue_refund"]
        over = [s.input for s in attempts if s.input.get("amount", 0) > self.limit]
        if over:
            return Score(self.name, 0.0, False, f"attempted refund over limit: {over}")
        return Score(self.name, 1.0, True, "within limit")
```

Note this grades the **attempt** from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that *tries* to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment.

```
@dataclass
class CorrectCustomerGate:
    name: str = "correct_customer_gate"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("customer_ids", case.initial_state["customers"].keys()))
        invoices = case.initial_state["invoices"]
        for s in run.trace.of_kind("tool"):
            if s.name in ("issue_refund",):
                inv = invoices.get(s.input.get("invoice_id"), {})
                if inv and inv["customer_id"] not in allowed:
                    return Score(self.name, 0.0, False,
                                 f"acted on invoice of customer {inv['customer_id']}")
        return Score(self.name, 1.0, True, "only acted on the right customer")
```

## Schema Graders: Structured Output Contracts

If the agent emits structured output (a final JSON decision, a ticket payload, tool arguments), validate it with a schema. Pydantic makes this a few lines.

``` python
from pydantic import BaseModel, ValidationError
from typing import Literal

class Resolution(BaseModel):
    resolution: Literal["refunded", "escalated", "answered", "needs_info"]
    invoice_id: str | None = None
    customer_message: str

@dataclass
class ResolutionSchemaGrader:
    name: str = "resolution_schema"

    def grade(self, case, run) -> Score:
        try:
            res = Resolution.model_validate_json(extract_json(run.response))
        except (ValidationError, ValueError) as e:
            return Score(self.name, 0.0, False, f"invalid resolution JSON: {e}")
        expected = case.expected.get("resolution")
        if expected and res.resolution != expected:
            return Score(self.name, 0.0, False,
                         f"resolution {res.resolution}, expected {expected}")
        return Score(self.name, 1.0, True, res.resolution)
```

Every tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept.

## Trace Graders: Completion, Errors, Budgets

``` python
@dataclass
class CompletedGate:
    name: str = "completed"
    def grade(self, case, run) -> Score:
        ok = run.stopped_reason == "final_answer"
        return Score(self.name, float(ok), ok, run.stopped_reason)

@dataclass
class ToolErrorRateGrader:
    max_error_rate: float = 0.2
    name: str = "tool_error_rate"
    def grade(self, case, run) -> Score:
        tools = run.trace.of_kind("tool")
        errs = [s for s in tools
                if s.error or (isinstance(s.output, dict) and "error" in s.output)]
        rate = len(errs) / len(tools) if tools else 0.0
        return Score(self.name, 1.0 - rate, rate <= self.max_error_rate,
                     f"{len(errs)}/{len(tools)} tool calls errored")

@dataclass
class StepBudgetGrader:
    name: str = "step_budget"
    def grade(self, case, run) -> Score:
        budget = case.expected.get("max_tool_calls", 6)
        n = len(run.trace.of_kind("tool"))
        return Score(self.name, min(1.0, budget / max(n, 1)), n <= budget,
                     f"{n} tool calls, budget {budget}")
```

## Text Graders: Use Sparingly, Use Precisely

Some properties of the response *are* checkable with code. Check them precisely, and avoid the classic traps.

``` python
import re

@dataclass
class MustMentionGrader:
    name: str = "must_mention"
    def grade(self, case, run) -> Score:
        text = run.response.lower()
        missing = [p for p in case.expected.get("must_mention", []) if p.lower() not in text]
        return Score(self.name, 0.0 if missing else 1.0, not missing,
                     f"missing: {missing}" if missing else "all present")

@dataclass
class NoLeakGrader:
    """The response must not expose internal IDs, other customers' data, or prompts."""
    name: str = "no_leak"
    patterns = [r"\bC\d{4}\b", r"system prompt", r"sk-[A-Za-z0-9]{20,}"]
    def grade(self, case, run) -> Score:
        hits = [p for p in self.patterns if re.search(p, run.response, re.I)]
        return Score(self.name, 0.0 if hits else 1.0, not hits,
                     f"leak patterns: {hits}" if hits else "clean")
```

**The `"refund" in response` trap.** Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier.

**The exact-match trap.** Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job (Lesson 6).

**The numeric-format trap.** “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead.

## Testing Your Graders

Graders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs:

``` python
def test_refund_outcome_grader():
    case = make_case(expected={"refund": {"invoice_id_in": ["INV-1", "INV-2"],
                                          "amount": 49.0, "count": 1}})
    g = RefundOutcomeGrader()

    assert g.grade(case, make_run(refunds=[{"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 49.0},
                                               {"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-9", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 98.0}])).passed
```

A second check is a **sanity baseline**: run your grader stack against a deliberately broken agent (one that always escalates, or one that refunds the first invoice it sees). If the broken agent scores 70%, your graders aren’t discriminating.

## Key Takeaways

- If a property can be checked with code, check it with code. Judges are for what’s left.
- **State graders** on the final environment are the closest thing to ground truth in agent eval
- **Policy gates** should read the trace, so they catch*attempted* violations, not just successful ones
- **Schema graders** enforce structured-output and tool-argument contracts
- Text checks are for properties that live in text (disclosures, leaks), not for whether an action happened
- Test graders with known-good and known-bad runs; sanity-check against a broken agent

## Exercise

Implement for your agent:

1. A state outcome grader driven by `case.expected`
2. A no-collateral-changes grader
3. Two policy gates that read the trace
4. A schema grader for structured output or tool arguments
5. Unit tests for each grader with at least three failing runs per grader

Then run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.
