# Testing AI agents is hard. I built a framework for it.

> Source: <https://dev.to/pawfromoz/testing-ai-agents-is-hard-i-built-a-framework-for-it-3hk0>
> Published: 2026-07-24 04:22:54+00:00

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly.

You don't find out until a user complains.

The problem: AI agents are non-deterministic. Traditional unit tests don't work. `expect(output).toBe("transfer to billing")`

fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.

I built [AgentSpec](https://github.com/Ozperium/agentspec) to fix this — a testing framework designed specifically for non-deterministic AI output.

Define tests in YAML:

```
name: "billing-agent-tests"
tests:
  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"

  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5
```

Run them:

```
npm install -g @ozperium/agentspec
agentspec init
agentspec run
```

Output:

```
AgentSpec v1.2.0

  ✓ routes billing question (312ms)
  ✓ handles expired token (891ms)
  ✗ response is on-topic
    Expected semantic similarity ≥ 0.5, got 0.31
    Input: "how do I reset my password?"
    Output: "Please contact support for account issues."

2 passed, 1 failed
```

Standard test assertions assume deterministic output. AI output isn't.

AgentSpec provides:

`contains`

/ `not_contains`

`contains_any`

/ `contains_all`

`semantically_similar`

`regex`

`tool_called`

`max_latency_ms`

`llm_judge`

```
agentspec run --ci   # exits 1 on failure, JUnit XML output
```

Drop it in GitHub Actions:

```
- name: Run AgentSpec
  run: agentspec run --ci
```

Now every PR that changes a prompt, model, or tool gets behavior regression tests.

Run with `--diff`

to compare against a previous baseline:

```
agentspec run --diff baseline.json
```

Shows exactly what changed between runs — useful when you swap models or update prompts.

Point it at any running agent endpoint:

```
agentspec run --endpoint http://localhost:3000/chat
```

No SDK integration required.

```
npm install -g @ozperium/agentspec
agentspec init
agentspec run
```

GitHub: [https://github.com/Ozperium/agentspec](https://github.com/Ozperium/agentspec)

*Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.*