How to Test AI Agents Without Calling More LLMs

A developer argues that AI-agent testing should separate orchestration correctness from language quality, using deterministic checks for tool order, retry limits, and state transitions instead of relying on expensive LLM judges. The approach uses dependency injection and scripted fakes to test execution contracts on every commit, reserving semantic evaluations for a smaller, calibrated workflow.

AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is stable enough for CI. LLM judges can be useful for genuinely semantic questions. They are a poor default for verifying tool order, retry limits, validation, budgets, state transitions, or error handling. Those behaviors have explicit contracts and can usually be tested with ordinary deterministic techniques. The most effective agent test suite separates orchestration correctness from language quality . Test the first on every commit with fakes, traces, and rules. Evaluate the second with a smaller, calibrated model-and-human workflow. An agent may produce variable language while still following a predictable execution contract. Useful deterministic checks include: None of these questions requires a second model. Most do not require a first model either. | Layer | Model access | Purpose | |---|---|---| | Pure unit tests | None | Routing, validation, budgeting, state reducers | | Orchestration tests | Scripted fake | Tool order, retries, fallbacks, termination | | Tool contract tests | None or sandbox | Schemas, timeouts, error mapping | | Trace replay tests | None | Regression checks against recorded execution metadata | | Model integration tests | Real model | Provider compatibility and a small set of end-to-end paths | | Semantic evaluations | Real model and/or human | Helpfulness, groundedness, tone, nuanced correctness | The lower layers should contain most of the suite. They are fast, reproducible, and actionable. The upper layers are valuable, but they should be deliberately smaller. Dependency injection makes the orchestration testable without network calls. type ModelReply = | { type: 'tool call'; tool: string; args: unknown } | { type: 'final'; text: string; usage: { input: number; output: number } }; interface ModelClient { generate input: { messages: Array<{ role: string; content: string } ; tools: string ; } : Promise<ModelReply ; } interface ToolClient { execute name: string, args: unknown : Promise<unknown ; } interface Clock { now : number; } Production code receives real implementations. Tests provide scripted implementations with known behavior. A scripted model returns a predefined sequence and fails if the agent makes an unexpected extra call. class ScriptedModel implements ModelClient { private index = 0; constructor private readonly replies: ModelReply {} async generate : Promise<ModelReply { const reply = this.replies this.index ; if reply { throw new Error Unexpected model call at index ${this.index} ; } this.index += 1; return structuredClone reply ; } assertConsumed : void { if this.index == this.replies.length { throw new Error Expected ${this.replies.length} model calls, received ${this.index} , ; } } } This fake does not imitate model intelligence. It controls the branch that the orchestration must handle. One script can request a tool and then return a final answer; another can repeatedly request an invalid tool call to verify termination. The trace should expose behavior without coupling tests to prose output. type TraceStep = { sequence: number; name: string; kind: 'run' | 'model' | 'tool' | 'policy' | 'fallback'; status: 'ok' | 'error' | 'blocked'; parentId: string | null; attempt?: number; metadata?: Record<string, string | number | boolean | null ; }; type AgentTrace = { status: 'ok' | 'error' | 'blocked'; steps: TraceStep ; }; Use a monotonic sequence assigned by the recorder for causal assertions. Do not infer retry behavior from consecutive names: parallel work can interleave events, and unrelated steps can separate attempts. Keep the trace schema stable and metadata-first. Tests should assert execution contracts, not depend on raw prompts or complete tool results. Small domain-specific helpers make failures easier to understand than generic array comparisons. function stepIndex trace: AgentTrace, name: string : number { return trace.steps.findIndex step = step.name === name ; } function expectStepBefore trace: AgentTrace, first: string, second: string, : void { const firstIndex = stepIndex trace, first ; const secondIndex = stepIndex trace, second ; if firstIndex < 0 throw new Error Missing trace step: ${first} ; if secondIndex < 0 throw new Error Missing trace step: ${second} ; if firstIndex = secondIndex { throw new Error Expected ${first} before ${second} ; } } function countSteps trace: AgentTrace, name: string : number { return trace.steps.filter step = step.name === name .length; } When operations run in parallel, assert parentage and required dependencies rather than total ordering. Two sibling tools may complete in either order and both be correct. js test 'retrieves policy before generating an answer', async = { const model = new ScriptedModel { type: 'final', text: 'Use the documented return process.', usage: { input: 420, output: 18 }, }, ; const { trace } = await runSupportAgent { model, tools: fakeTools, clock: fakeClock }, 'How do returns work?', ; expectStepBefore trace, 'retrieve policy', 'generate answer' ; expect countSteps trace, 'retrieve policy' .toBe 1 ; model.assertConsumed ; } ; The test says exactly what failed: a missing step, an invalid dependency order, or an unexpected model call. No quality score is needed. js test 'stops a tool after two failed attempts', async = { const tools = new FakeTools { lookup invoice: new TimeoutError , new TimeoutError , { invoiceId: 'should-never-be-returned' }, , } ; const { trace } = await runInvoiceAgent { model: scriptedToolCaller, tools, clock: fakeClock }, 'Find the latest invoice', ; const attempts = trace.steps.filter step = step.name === 'lookup invoice', ; expect attempts.map step = step.attempt .toEqual 1, 2 ; expect tools.calls 'lookup invoice' .toHaveLength 2 ; expect trace.status .toBe 'error' ; } ; This catches an unbounded retry loop without waiting for a real service or model. A blocked request should produce no downstream model or tool activity. js test 'blocks unauthorized export before external calls', async = { const { trace } = await runExportAgent { model: modelThatMustNotRun, tools: toolsThatMustNotRun, clock: fakeClock }, { action: 'export all accounts', authorized: false }, ; expect trace.status .toBe 'blocked' ; expect stepIndex trace, 'authorize export' .toBeGreaterThanOrEqual 0 ; expect trace.steps.some step = step.kind === 'model' .toBe false ; expect trace.steps.some step = step.kind === 'tool' .toBe false ; } ; The test uses structured input instead of asking a model to recognize a particular malicious phrase. A separate security suite can test prompt-injection resistance with representative text fixtures. Wall-clock assertions are often flaky under CI load. Inject a fake clock or scheduler and advance it deliberately. class FakeClock implements Clock { private value = 0; now : number { return this.value; } advance ms: number : void { this.value += ms; } } test 'uses fallback after the configured timeout', async = { const clock = new FakeClock ; const tools = new FakeTools { primary search: new TimeoutError } ; const { trace } = await runSearchAgent { model: scriptedToolCaller, tools, clock }, 'pricing', ; expect stepIndex trace, 'fallback search' .toBeGreaterThanOrEqual 0 ; } ; Keep a small number of real timing tests for integration boundaries. Do not make every unit test depend on an overloaded runner finishing within an arbitrary number of milliseconds. Model fakes can return explicit usage values. That lets you verify budget logic without paying for tokens. js test 'stops before exceeding the session token budget', async = { const model = new ScriptedModel { type: 'final', text: 'first', usage: { input: 700, output: 200 } }, { type: 'final', text: 'second', usage: { input: 700, output: 200 } }, ; const { trace } = await runBudgetedAgent { model, tools: fakeTools, clock: fakeClock, maxTokens: 1 200 }, 'continue', ; expect trace.steps.some step = step.name === 'token budget block' .toBe true ; expect countSteps trace, 'model call' .toBe 1 ; } ; This proves the enforcement behavior. A separate provider integration test can verify that production usage fields are mapped correctly into the internal token model. Recorded metadata traces can drive tests without replaying sensitive content or calling a model. Store a compact fixture describing model decisions, tool outcomes, usage, and expected invariants. Avoid treating the entire trace as a snapshot that must match byte for byte. IDs, timestamps, and harmless implementation details make snapshots brittle. Normalize the trace and assert durable properties: Update a fixture only when the behavioral contract intentionally changes. Agents fail at boundaries. Include fixtures for: Verify that the original error category is preserved, cleanup runs, and no further model or tool work occurs after a terminal state. Deterministic tests cannot prove that an answer is clear, grounded, polite, or semantically correct. Real-model evaluation remains useful for those questions. Use a curated dataset and an explicit rubric. Calibrate judge scores against human labels, track disagreement, and review threshold failures. Because model behavior can change, compare distributions and trends rather than treating one score from one run as an unquestionable fact. A practical schedule is: | Trigger | Recommended checks | |---|---| | Local save | Unit and scripted orchestration tests | | Every commit | Trace invariants, policy, retries, budgets, tool contracts | | Pull request | Deterministic suite plus a small real-model smoke set | | Nightly or weekly | Larger semantic evaluation and regression trends | | Release candidate | Human review of high-risk or changed workflows | Teams with stricter cost or reliability requirements can move real-model tests out of pull requests entirely. The important point is to make that trade-off explicit. Agent output is probabilistic, but agent architecture does not need to be opaque. Validation, tool access, retries, fallbacks, budgets, state transitions, and trace structure can all have deterministic contracts. Put those contracts behind injectable interfaces, drive them with scripted dependencies, and assert behavior through a stable metadata trace. Save real models and LLM judges for the semantic questions that ordinary code cannot answer. The next article will turn these ideas into fast CI quality gates with reusable trace rules, baseline comparison, and actionable failure reports.