How to Test AI Agents Without Calling More LLMs

AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is stable enough for CI.

LLM judges can be useful for genuinely semantic questions. They are a poor default for verifying tool order, retry limits, validation, budgets, state transitions, or error handling. Those behaviors have explicit contracts and can usually be tested with ordinary deterministic techniques.

The most effective agent test suite separates orchestration correctness from language quality. Test the first on every commit with fakes, traces, and rules. Evaluate the second with a smaller, calibrated model-and-human workflow.

An agent may produce variable language while still following a predictable execution contract. Useful deterministic checks include:

None of these questions requires a second model. Most do not require a first model either.

The lower layers should contain most of the suite. They are fast, reproducible, and actionable. The upper layers are valuable, but they should be deliberately smaller.

Dependency injection makes the orchestration testable without network calls.

type ModelReply =
  | { type: 'tool_call'; tool: string; args: unknown }
  | { type: 'final'; text: string; usage: { input: number; output: number } };

interface ModelClient {
  generate(input: {
    messages: Array<{ role: string; content: string }>;
    tools: string[];
  }): Promise<ModelReply>;
}

interface ToolClient {
  execute(name: string, args: unknown): Promise<unknown>;
}

interface Clock {
  now(): number;
}

Production code receives real implementations. Tests provide scripted implementations with known behavior.

A scripted model returns a predefined sequence and fails if the agent makes an unexpected extra call.

class ScriptedModel implements ModelClient {
  private index = 0;

  constructor(private readonly replies: ModelReply[]) {}

  async generate(): Promise<ModelReply> {
    const reply = this.replies[this.index];
    if (!reply) {
      throw new Error(`Unexpected model call at index ${this.index}`);
    }

    this.index += 1;
    return structuredClone(reply);
  }

  assertConsumed(): void {
    if (this.index !== this.replies.length) {
      throw new Error(
        `Expected ${this.replies.length} model calls, received ${this.index}`,
      );
    }
  }
}

This fake does not imitate model intelligence. It controls the branch that the orchestration must handle. One script can request a tool and then return a final answer; another can repeatedly request an invalid tool call to verify termination.

The trace should expose behavior without coupling tests to prose output.

type TraceStep = {
  sequence: number;
  name: string;
  kind: 'run' | 'model' | 'tool' | 'policy' | 'fallback';
  status: 'ok' | 'error' | 'blocked';
  parentId: string | null;
  attempt?: number;
  metadata?: Record<string, string | number | boolean | null>;
};

type AgentTrace = {
  status: 'ok' | 'error' | 'blocked';
  steps: TraceStep[];
};

Use a monotonic sequence assigned by the recorder for causal assertions. Do not infer retry behavior from consecutive names: parallel work can interleave events, and unrelated steps can separate attempts.

Keep the trace schema stable and metadata-first. Tests should assert execution contracts, not depend on raw prompts or complete tool results.

Small domain-specific helpers make failures easier to understand than generic array comparisons.

function stepIndex(trace: AgentTrace, name: string): number {
  return trace.steps.findIndex((step) => step.name === name);
}

function expectStepBefore(
  trace: AgentTrace,
  first: string,
  second: string,
): void {
  const firstIndex = stepIndex(trace, first);
  const secondIndex = stepIndex(trace, second);

  if (firstIndex < 0) throw new Error(`Missing trace step: ${first}`);
  if (secondIndex < 0) throw new Error(`Missing trace step: ${second}`);
  if (firstIndex >= secondIndex) {
    throw new Error(`Expected ${first} before ${second}`);
  }
}

function countSteps(trace: AgentTrace, name: string): number {
  return trace.steps.filter((step) => step.name === name).length;
}

When operations run in parallel, assert parentage and required dependencies rather than total ordering. Two sibling tools may complete in either order and both be correct.

test('retrieves policy before generating an answer', async () => {
  const model = new ScriptedModel([
    {
      type: 'final',
      text: 'Use the documented return process.',
      usage: { input: 420, output: 18 },
    },
  ]);

  const { trace } = await runSupportAgent(
    { model, tools: fakeTools, clock: fakeClock },
    'How do returns work?',
  );

  expectStepBefore(trace, 'retrieve_policy', 'generate_answer');
  expect(countSteps(trace, 'retrieve_policy')).toBe(1);
  model.assertConsumed();
});

The test says exactly what failed: a missing step, an invalid dependency order, or an unexpected model call. No quality score is needed.

test('stops a tool after two failed attempts', async () => {
  const tools = new FakeTools({
    lookup_invoice: [
      new TimeoutError(),
      new TimeoutError(),
      { invoiceId: 'should-never-be-returned' },
    ],
  });

  const { trace } = await runInvoiceAgent(
    { model: scriptedToolCaller, tools, clock: fakeClock },
    'Find the latest invoice',
  );

  const attempts = trace.steps.filter(
    (step) => step.name === 'lookup_invoice',
  );

  expect(attempts.map((step) => step.attempt)).toEqual([1, 2]);
  expect(tools.calls('lookup_invoice')).toHaveLength(2);
  expect(trace.status).toBe('error');
});

This catches an unbounded retry loop without waiting for a real service or model.

A blocked request should produce no downstream model or tool activity.

test('blocks unauthorized export before external calls', async () => {
  const { trace } = await runExportAgent(
    { model: modelThatMustNotRun, tools: toolsThatMustNotRun, clock: fakeClock },
    { action: 'export_all_accounts', authorized: false },
  );

  expect(trace.status).toBe('blocked');
  expect(stepIndex(trace, 'authorize_export')).toBeGreaterThanOrEqual(0);
  expect(trace.steps.some((step) => step.kind === 'model')).toBe(false);
  expect(trace.steps.some((step) => step.kind === 'tool')).toBe(false);
});

The test uses structured input instead of asking a model to recognize a particular malicious phrase. A separate security suite can test prompt-injection resistance with representative text fixtures.

Wall-clock assertions are often flaky under CI load. Inject a fake clock or scheduler and advance it deliberately.

class FakeClock implements Clock {
  private value = 0;

  now(): number {
    return this.value;
  }

  advance(ms: number): void {
    this.value += ms;
  }
}

test('uses fallback after the configured timeout', async () => {
  const clock = new FakeClock();
  const tools = new FakeTools({ primary_search: [new TimeoutError()] });

  const { trace } = await runSearchAgent(
    { model: scriptedToolCaller, tools, clock },
    'pricing',
  );

  expect(stepIndex(trace, 'fallback_search')).toBeGreaterThanOrEqual(0);
});

Keep a small number of real timing tests for integration boundaries. Do not make every unit test depend on an overloaded runner finishing within an arbitrary number of milliseconds.

Model fakes can return explicit usage values. That lets you verify budget logic without paying for tokens.

test('stops before exceeding the session token budget', async () => {
  const model = new ScriptedModel([
    { type: 'final', text: 'first', usage: { input: 700, output: 200 } },
    { type: 'final', text: 'second', usage: { input: 700, output: 200 } },
  ]);

  const { trace } = await runBudgetedAgent(
    { model, tools: fakeTools, clock: fakeClock, maxTokens: 1_200 },
    'continue',
  );

  expect(trace.steps.some((step) => step.name === 'token_budget_block')).toBe(true);
  expect(countSteps(trace, 'model_call')).toBe(1);
});

This proves the enforcement behavior. A separate provider integration test can verify that production usage fields are mapped correctly into the internal token model.

Recorded metadata traces can drive tests without replaying sensitive content or calling a model. Store a compact fixture describing model decisions, tool outcomes, usage, and expected invariants.

Avoid treating the entire trace as a snapshot that must match byte for byte. IDs, timestamps, and harmless implementation details make snapshots brittle. Normalize the trace and assert durable properties:

Update a fixture only when the behavioral contract intentionally changes.

Agents fail at boundaries. Include fixtures for:

Verify that the original error category is preserved, cleanup runs, and no further model or tool work occurs after a terminal state.

Deterministic tests cannot prove that an answer is clear, grounded, polite, or semantically correct. Real-model evaluation remains useful for those questions.

Use a curated dataset and an explicit rubric. Calibrate judge scores against human labels, track disagreement, and review threshold failures. Because model behavior can change, compare distributions and trends rather than treating one score from one run as an unquestionable fact.

A practical schedule is:

Teams with stricter cost or reliability requirements can move real-model tests out of pull requests entirely. The important point is to make that trade-off explicit.

Agent output is probabilistic, but agent architecture does not need to be opaque. Validation, tool access, retries, fallbacks, budgets, state transitions, and trace structure can all have deterministic contracts.

Put those contracts behind injectable interfaces, drive them with scripted dependencies, and assert behavior through a stable metadata trace. Save real models and LLM judges for the semantic questions that ordinary code cannot answer.

The next article will turn these ideas into fast CI quality gates with reusable trace rules, baseline comparison, and actionable failure reports.