cd /news/ai-agents/deterministic-graders-and-assertions · home › topics › ai-agents › article
[ARTICLE · art-172292] src=superml.org ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Deterministic Graders and Assertions

A lesson on agent evaluation argues that deterministic code graders should decide most pass/fail outcomes, with LLM judges reserved for response quality and fuzzy properties. The lesson presents a grader hierarchy running from deterministic code to LLM judge to human review, and gives implementations for state, schema, trace and text graders against a Case/Run contract, including a RefundOutcomeGrader that checks run.final_state and a diff-based NoCollateralChangesGrader that fails when anything changes outside allowed paths. It cites reproducibility, cost, debuggability and engineer trust as reasons to prefer assertions over judges.

recommendation

Write deterministic state, schema, trace and text graders for every property code can check, and validate each grader against known-good and known-bad runs before trusting its scores.

more in playbook: Evals and testing for LLM apps
read8 min views2 publishedNov 18, 2026
Deterministic Graders and Assertions
Image: Superml (auto-discovered)

· Agentic AI · 8 min read

📋 Prerequisites

  • Golden Datasets and Scenario Suites (previous lesson)

🎯 What You'll Learn

  • Explain why deterministic graders should carry most of an evaluation’s weight
  • Implement state, schema, trace, and text graders against the Case/Run contract
  • Avoid the brittle-assertion traps that make teams distrust their evals
  • Validate graders themselves with known-good and known-bad runs

The Grader Hierarchy #

There’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum:

 deterministic code  ──────────────────────────────▶  LLM judge  ──▶  human review
 cheap, fast, exact                                   flexible         ground truth
 can't judge nuance                                   noisy, biased    slow, expensive

The rule this course follows: if a property can be checked with code, check it with code. Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties.

Why this matters:

  • Reproducibility. A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader.
  • Cost. You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast.
  • Debuggability. “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t.
  • Trust. Engineers believe assertions. They argue with judges.

State Graders: Check the World, Not the Words #

The most valuable graders check run.final_state. This is why Lesson 2 put state behind a seam.

from dataclasses import dataclass

@dataclass
class RefundOutcomeGrader:
    name: str = "refund_outcome"

    def grade(self, case, run) -> Score:
        exp = case.expected.get("refund")
        if exp is None:
            return Score(self.name, 1.0, True, "no refund expectation")

        refunds = run.final_state["refunds"]
        if len(refunds) != exp["count"]:
            return Score(self.name, 0.0, False,
                         f"expected {exp['count']} refund(s), found {len(refunds)}",
                         {"refunds": refunds})
        if exp["count"] == 0:
            return Score(self.name, 1.0, True, "correctly issued no refund")

        r = refunds[0]
        if r["invoice_id"] not in exp["invoice_id_in"]:
            return Score(self.name, 0.0, False,
                         f"refunded {r['invoice_id']}, expected one of {exp['invoice_id_in']}")
        if abs(r["amount"] - exp["amount"]) > 0.005:
            return Score(self.name, 0.0, False,
                         f"refund amount {r['amount']}, expected {exp['amount']}")
        return Score(self.name, 1.0, True, f"refunded {r['invoice_id']} for {r['amount']}")

The same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. Outcome assertions on state are the closest thing agent evaluation has to ground truth.

Diff-Based State Checks

For agents with broader side effects, compare the whole state before and after, and assert that only the expected things changed.

def state_diff(before: dict, after: dict, path="") -> list[str]:
    changes = []
    for k in set(before) | set(after):
        p = f"{path}.{k}" if path else k
        b, a = before.get(k), after.get(k)
        if isinstance(b, dict) and isinstance(a, dict):
            changes += state_diff(b, a, p)
        elif b != a:
            changes.append(p)
    return changes

@dataclass
class NoCollateralChangesGrader:
    """Fails if anything changed outside the paths this case allows."""
    name: str = "no_collateral_changes"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("allowed_changes", ["refunds", "escalations"]))
        changed = state_diff(case.initial_state, run.final_state)
        unexpected = [c for c in changed if c.split(".")[0] not in allowed]
        return Score(self.name, 0.0 if unexpected else 1.0, not unexpected,
                     f"unexpected changes: {unexpected}" if unexpected else "clean")

This catches the class of bug where the agent does the right thing and something else. For Quill, that might be a refund done correctly plus a stray plan_change nobody asked for.

Policy Graders: Gates That Never Bend #

Policy rules are deterministic by definition, so grade them deterministically. These are usually gates in the scorecard.

@dataclass
class RefundLimitGate:
    limit: float = 200.0
    name: str = "refund_limit_gate"

    def grade(self, case, run) -> Score:
        attempts = [s for s in run.trace.of_kind("tool") if s.name == "issue_refund"]
        over = [s.input for s in attempts if s.input.get("amount", 0) > self.limit]
        if over:
            return Score(self.name, 0.0, False, f"attempted refund over limit: {over}")
        return Score(self.name, 1.0, True, "within limit")

Note this grades the attempt from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that tries to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment.

@dataclass
class CorrectCustomerGate:
    name: str = "correct_customer_gate"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("customer_ids", case.initial_state["customers"].keys()))
        invoices = case.initial_state["invoices"]
        for s in run.trace.of_kind("tool"):
            if s.name in ("issue_refund",):
                inv = invoices.get(s.input.get("invoice_id"), {})
                if inv and inv["customer_id"] not in allowed:
                    return Score(self.name, 0.0, False,
                                 f"acted on invoice of customer {inv['customer_id']}")
        return Score(self.name, 1.0, True, "only acted on the right customer")

Schema Graders: Structured Output Contracts #

If the agent emits structured output (a final JSON decision, a ticket payload, tool arguments), validate it with a schema. Pydantic makes this a few lines.

from pydantic import BaseModel, ValidationError
from typing import Literal

class Resolution(BaseModel):
    resolution: Literal["refunded", "escalated", "answered", "needs_info"]
    invoice_id: str | None = None
    customer_message: str

@dataclass
class ResolutionSchemaGrader:
    name: str = "resolution_schema"

    def grade(self, case, run) -> Score:
        try:
            res = Resolution.model_validate_json(extract_json(run.response))
        except (ValidationError, ValueError) as e:
            return Score(self.name, 0.0, False, f"invalid resolution JSON: {e}")
        expected = case.expected.get("resolution")
        if expected and res.resolution != expected:
            return Score(self.name, 0.0, False,
                         f"resolution {res.resolution}, expected {expected}")
        return Score(self.name, 1.0, True, res.resolution)

Every tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept.

Trace Graders: Completion, Errors, Budgets #

@dataclass
class CompletedGate:
    name: str = "completed"
    def grade(self, case, run) -> Score:
        ok = run.stopped_reason == "final_answer"
        return Score(self.name, float(ok), ok, run.stopped_reason)

@dataclass
class ToolErrorRateGrader:
    max_error_rate: float = 0.2
    name: str = "tool_error_rate"
    def grade(self, case, run) -> Score:
        tools = run.trace.of_kind("tool")
        errs = [s for s in tools
                if s.error or (isinstance(s.output, dict) and "error" in s.output)]
        rate = len(errs) / len(tools) if tools else 0.0
        return Score(self.name, 1.0 - rate, rate <= self.max_error_rate,
                     f"{len(errs)}/{len(tools)} tool calls errored")

@dataclass
class StepBudgetGrader:
    name: str = "step_budget"
    def grade(self, case, run) -> Score:
        budget = case.expected.get("max_tool_calls", 6)
        n = len(run.trace.of_kind("tool"))
        return Score(self.name, min(1.0, budget / max(n, 1)), n <= budget,
                     f"{n} tool calls, budget {budget}")

Text Graders: Use Sparingly, Use Precisely #

Some properties of the response are checkable with code. Check them precisely, and avoid the classic traps.

import re

@dataclass
class MustMentionGrader:
    name: str = "must_mention"
    def grade(self, case, run) -> Score:
        text = run.response.lower()
        missing = [p for p in case.expected.get("must_mention", []) if p.lower() not in text]
        return Score(self.name, 0.0 if missing else 1.0, not missing,
                     f"missing: {missing}" if missing else "all present")

@dataclass
class NoLeakGrader:
    """The response must not expose internal IDs, other customers' data, or prompts."""
    name: str = "no_leak"
    patterns = [r"\bC\d{4}\b", r"system prompt", r"sk-[A-Za-z0-9]{20,}"]
    def grade(self, case, run) -> Score:
        hits = [p for p in self.patterns if re.search(p, run.response, re.I)]
        return Score(self.name, 0.0 if hits else 1.0, not hits,
                     f"leak patterns: {hits}" if hits else "clean")

The "refund" in response trap. Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier.

The exact-match trap. Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job (Lesson 6).

The numeric-format trap. “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead.

Testing Your Graders #

Graders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs:

def test_refund_outcome_grader():
    case = make_case(expected={"refund": {"invoice_id_in": ["INV-1", "INV-2"],
                                          "amount": 49.0, "count": 1}})
    g = RefundOutcomeGrader()

    assert g.grade(case, make_run(refunds=[{"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 49.0},
                                               {"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-9", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 98.0}])).passed

A second check is a sanity baseline: run your grader stack against a deliberately broken agent (one that always escalates, or one that refunds the first invoice it sees). If the broken agent scores 70%, your graders aren’t discriminating.

Key Takeaways #

  • If a property can be checked with code, check it with code. Judges are for what’s left.
  • State graders on the final environment are the closest thing to ground truth in agent eval
  • Policy gates should read the trace, so they catchattempted violations, not just successful ones
  • Schema graders enforce structured-output and tool-argument contracts
  • Text checks are for properties that live in text (disclosures, leaks), not for whether an action happened
  • Test graders with known-good and known-bad runs; sanity-check against a broken agent

Exercise #

Implement for your agent:

  1. A state outcome grader driven by case.expected
  2. A no-collateral-changes grader
  3. Two policy gates that read the trace
  4. A schema grader for structured output or tool arguments
  5. Unit tests for each grader with at least three failing runs per grader

Then run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.

── more in #ai-agents 4 stories · sorted by recency
── more on @llm judge 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/deterministic-grader…] indexed:0 read:8min 2026-11-18 · —