How to Evaluate AI Agents for Production: Moving Beyond LLM-as-a-Judge

A developer published a five-part series arguing that LLM-as-a-judge evaluation fails to protect production AI agents, because judges grade an agent's prose claims rather than the actual environment changes it makes. The writeup models multi-step reliability as p^k, showing that a 90% per-step success rate yields only about a 35% chance of completing a 10-step workflow, and calls for state-based, deterministic evaluation at each transition instead of transcript scoring.

This is Part 1 of a 5-part series on evaluating and hardening production AI agents. See also: Part 2: Deterministic State Evaluations https://employabledatascientist.com/blog/deterministic-state-evaluations-ai-agent-tool-calls/ , Part 3: Statistical Rigor in AI Evals https://employabledatascientist.com/blog/statistical-rigor-in-ai-evals-confidence-intervals-sample-sizing/ , Part 4: Curating Golden Trajectories https://employabledatascientist.com/blog/curating-golden-trajectories-adversarial-agent-red-teaming/ , and Part 5: Building an Agent CI/CD Pipeline https://employabledatascientist.com/blog/building-agent-evaluation-cicd-pipeline-regression-gates/ . Most AI agent demos look magical until they face unconstrained user inputs and real environment states. In a controlled demo, an agent receives a clean prompt, calls two tools, and outputs a coherent final response. In production, teams encounter the reliability cliff: the agent gets stuck in infinite retry loops, hallucinates schema parameters, mutates production databases unexpectedly, or drifts into degenerative completions. To solve this, engineering teams frequently reach for the industry default: LLM-as-a-judge . They pipe the agent's multi-turn transcript into GPT-4 or Claude, provide a rubric prompt, and ask the model to rate task completion on a scale from 1 to 5. This approach creates a dangerous illusion of measurement. It gives leadership a plausible-looking dashboard while leaving production systems entirely unprotected against regression. Here is why LLM-as-a-judge fails to protect production agents, the compounding mathematics underneath multi-step failure, and the state-based evaluation architecture required to deploy agents with confidence. The fundamental difference between a single-turn completion such as summarization or question-answering and an autonomous agent is sequential dependency . In a multi-turn agent, every intermediate tool execution changes the environment and feeds into the next turn's context window. If Turn 1 retrieves an incorrect ID, Turn 2 operates on bad data, Turn 3 issues a corrupt database mutation, and Turn 4 hallucinates an explanation to justify the discrepancy. Consider an agent task requiring $k$ discrete steps. If each step has an independent success probability $p$, the end-to-end task success probability $P \text{task} $ follows: $$P \text{task} = p^k$$ If your underlying foundation model is highly capable—say, $p = 0.90$ 90% per-step reliability —the compound trajectory success decays rapidly as task complexity scales: A system where individual steps look stellar during ad-hoc testing fails two out of three times when executing 10-step workflows. Worse, steps are rarely independent. A minor error early in a trajectory increases the entropy of the agent's context window, degrading subsequent reasoning performance. Without deterministic guards at each transition, error cascading is mathematically inevitable. Using a language model to grade multi-turn transcripts introduces three major measurement errors: Language models evaluate text tokens; they do not execute environments. If an agent reports: "I have reconciled invoice 8492 and updated the ERP ledger status to settled." An LLM judge reads this transcript and gives it a score of 5/5 for helpfulness, tone, and apparent completion. But did the invoice actually update in the database? Did it credit the right account balance? Did it violate foreign key constraints? The judge cannot know because it only inspects the prose claim, not the environment delta. Grading agents on what they say rather than what they changed is the primary source of unmonitored production outages. Empirical evaluation benchmarks repeatedly reveal systematic structural biases in frontier models acting as evaluators: When you evaluate a non-deterministic agent with a non-deterministic judge, you double your measurement variance. If an evaluation score drops from 4.2 to 3.8 on a nightly run, your team cannot immediately tell whether: An evaluation harness that engineering teams cannot reproduce is an evaluation harness they will eventually ignore. To build an evaluation suite that survives engineering inspection, we invert the paradigm: Evaluate state first, tool execution second, and prose last. ┌────────────────────────────────────────────────────────┐ │ Evaluation Pyramid for Agents │ ├────────────────────────────────────────────────────────┤ │ Layer 3: Semantic / Prose Quality │ │ Embeddings, semantic similarity, tone guards │ ├────────────────────────────────────────────────────────┤ │ Layer 2: Trajectory & Tool Efficiency │ │ Schema validation, step counts, token budget, loops │ ├────────────────────────────────────────────────────────┤ │ Layer 1: Deterministic State Assertions │ │ Database deltas, sandbox files, mock API contracts │ └────────────────────────────────────────────────────────┘ Before an agent executes a test case, snapshot the environment. After the agent halts, assert exact environment state: These assertions are deterministic, instant, free of token costs, and 100% reproducible. Even if the final state matches, how the agent got there matters for latency, cost, and reliability: Only after Layers 1 and 2 pass do you evaluate prose. Instead of open-ended 1-to-5 scoring, use narrow, single-question binary assertions: Here is a minimal, robust harness showing how to evaluate an agent against deterministic environment assertions: python from dataclasses import dataclass from typing import Any, Callable, Dict, List import pytest @dataclass class EvalTestCase: task id: str user prompt: str initial state: Dict str, Any expected state assertion: Callable Dict str, Any , bool max steps: int = 5 max cost usd: float = 0.05 def test agent reconciliation flow : Setup test fixture with isolated mock state mock db = {"invoices": {"inv 101": {"amount": 250.0, "status": "pending"}}} test case = EvalTestCase task id="reconcile paid invoice", user prompt="Reconcile payment of $250.00 for invoice inv 101.", initial state=mock db, expected state assertion=lambda state: state "invoices" "inv 101" "status" == "settled" and state "invoices" "inv 101" "amount" == 250.0 , max steps=3 Execute agent against the mock environment trajectory = run agent test case.user prompt, state=mock db 1. Assert deterministic final state assert test case.expected state assertion mock db , f"State assertion failed for {test case.task id}. Current DB: {mock db}" 2. Assert trajectory efficiency constraints assert len trajectory.steps <= test case.max steps, f"Agent exceeded step budget: {len trajectory.steps } {test case.max steps}" 3. Assert zero schema validation errors during tool calls for step in trajectory.steps: assert step.schema valid is True, f"Malformed tool call in turn {step.turn id}" When an engineer breaks agent functionality in a pull request, this harness outputs an exact stack trace showing the mismatched database row or the malformed schema parameter, rather than a subjective LLM judge score dropping by 0.2 points. In Part 2 , we examine the exact implementation of Deterministic State Evaluations : how to mock sandboxed external tools, validate JSON Schema bounds, and capture before-and-after database states automatically. If your team is building an AI product or agentic system and struggling with silent failures, high latency, or unreliable tool calling: We embed directly with AI engineering teams as a Hacker-in-Residence https://employabledatascientist.com/services/hacker-in-residence/ . We architect custom evaluation harnesses, red-team non-deterministic failure modes, and build the CI/CD regression gates your team needs to ship with conviction.