How to Evaluate AI Agents for Production: Moving Beyond LLM-as-a-Judge

This is Part 1 of a 5-part series on evaluating and hardening production AI agents. See also: Part 2: Deterministic State Evaluations, Part 3: Statistical Rigor in AI Evals, Part 4: Curating Golden Trajectories, and Part 5: Building an Agent CI/CD Pipeline.

Most AI agent demos look magical until they face unconstrained user inputs and real environment states.

In a controlled demo, an agent receives a clean prompt, calls two tools, and outputs a coherent final response. In production, teams encounter the reliability cliff: the agent gets stuck in infinite retry loops, hallucinates schema parameters, mutates production databases unexpectedly, or drifts into degenerative completions.

To solve this, engineering teams frequently reach for the industry default: LLM-as-a-judge. They pipe the agent's multi-turn transcript into GPT-4 or Claude, provide a rubric prompt, and ask the model to rate task completion on a scale from 1 to 5.

This approach creates a dangerous illusion of measurement. It gives leadership a plausible-looking dashboard while leaving production systems entirely unprotected against regression.

Here is why LLM-as-a-judge fails to protect production agents, the compounding mathematics underneath multi-step failure, and the state-based evaluation architecture required to deploy agents with confidence.

The fundamental difference between a single-turn completion (such as summarization or question-answering) and an autonomous agent is sequential dependency.

In a multi-turn agent, every intermediate tool execution changes the environment and feeds into the next turn's context window. If Turn 1 retrieves an incorrect ID, Turn 2 operates on bad data, Turn 3 issues a corrupt database mutation, and Turn 4 hallucinates an explanation to justify the discrepancy.

Consider an agent task requiring $k$ discrete steps. If each step has an independent success probability $p$, the end-to-end task success probability $P(\text{task})$ follows:

$$P(\text{task}) = p^k$$

If your underlying foundation model is highly capable—say, $p = 0.90$ (90% per-step reliability)—the compound trajectory success decays rapidly as task complexity scales:

A system where individual steps look stellar during ad-hoc testing fails two out of three times when executing 10-step workflows.

Worse, steps are rarely independent. A minor error early in a trajectory increases the entropy of the agent's context window, degrading subsequent reasoning performance. Without deterministic guards at each transition, error cascading is mathematically inevitable.

Using a language model to grade multi-turn transcripts introduces three major measurement errors:

Language models evaluate text tokens; they do not execute environments.

If an agent reports:

"I have reconciled invoice #8492 and updated the ERP ledger status to settled."

An LLM judge reads this transcript and gives it a score of 5/5 for helpfulness, tone, and apparent completion. But did the invoice actually update in the database? Did it credit the right account balance? Did it violate foreign key constraints?

The judge cannot know because it only inspects the prose claim, not the environment delta. Grading agents on what they say rather than what they changed is the primary source of unmonitored production outages.

Empirical evaluation benchmarks repeatedly reveal systematic structural biases in frontier models acting as evaluators:

When you evaluate a non-deterministic agent with a non-deterministic judge, you double your measurement variance. If an evaluation score drops from 4.2 to 3.8 on a nightly run, your team cannot immediately tell whether:

An evaluation harness that engineering teams cannot reproduce is an evaluation harness they will eventually ignore.

To build an evaluation suite that survives engineering inspection, we invert the paradigm: Evaluate state first, tool execution second, and prose last.

┌────────────────────────────────────────────────────────┐
│               Evaluation Pyramid for Agents             │
├────────────────────────────────────────────────────────┤
│ Layer 3: Semantic / Prose Quality                      │
│ (Embeddings, semantic similarity, tone guards)         │
├────────────────────────────────────────────────────────┤
│ Layer 2: Trajectory & Tool Efficiency                  │
│ (Schema validation, step counts, token budget, loops)  │
├────────────────────────────────────────────────────────┤
│ Layer 1: Deterministic State Assertions                │
│ (Database deltas, sandbox files, mock API contracts)   │
└────────────────────────────────────────────────────────┘

Before an agent executes a test case, snapshot the environment. After the agent halts, assert exact environment state:

These assertions are deterministic, instant, free of token costs, and 100% reproducible.

Even if the final state matches, how the agent got there matters for latency, cost, and reliability:

Only after Layers 1 and 2 pass do you evaluate prose. Instead of open-ended 1-to-5 scoring, use narrow, single-question binary assertions:

Here is a minimal, robust harness showing how to evaluate an agent against deterministic environment assertions:

from dataclasses import dataclass
from typing import Any, Callable, Dict, List
import pytest

@dataclass
class EvalTestCase:
    task_id: str
    user_prompt: str
    initial_state: Dict[str, Any]
    expected_state_assertion: Callable[[Dict[str, Any]], bool]
    max_steps: int = 5
    max_cost_usd: float = 0.05

def test_agent_reconciliation_flow():
    mock_db = {"invoices": {"inv_101": {"amount": 250.0, "status": "pending"}}}

    test_case = EvalTestCase(
        task_id="reconcile_paid_invoice",
        user_prompt="Reconcile payment of $250.00 for invoice inv_101.",
        initial_state=mock_db,
        expected_state_assertion=lambda state: (
            state["invoices"]["inv_101"]["status"] == "settled"
            and state["invoices"]["inv_101"]["amount"] == 250.0
        ),
        max_steps=3
    )

    trajectory = run_agent(test_case.user_prompt, state=mock_db)

    assert test_case.expected_state_assertion(mock_db), (
        f"State assertion failed for {test_case.task_id}. Current DB: {mock_db}"
    )

    assert len(trajectory.steps) <= test_case.max_steps, (
        f"Agent exceeded step budget: {len(trajectory.steps)} > {test_case.max_steps}"
    )

    for step in trajectory.steps:
        assert step.schema_valid is True, f"Malformed tool call in turn {step.turn_id}"

When an engineer breaks agent functionality in a pull request, this harness outputs an exact stack trace showing the mismatched database row or the malformed schema parameter, rather than a subjective LLM judge score dropping by 0.2 points.

In Part 2, we examine the exact implementation of Deterministic State Evaluations: how to mock sandboxed external tools, validate JSON Schema bounds, and capture before-and-after database states automatically.

If your team is building an AI product or agentic system and struggling with silent failures, high latency, or unreliable tool calling:

We embed directly with AI engineering teams as a Hacker-in-Residence. We architect custom evaluation harnesses, red-team non-deterministic failure modes, and build the CI/CD regression gates your team needs to ship with conviction.