Testing AI agents is hard. I built a framework for it.

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly.

You don't find out until a user complains.

The problem: AI agents are non-deterministic. Traditional unit tests don't work. expect(output).toBe("transfer to billing")

fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.

I built AgentSpec to fix this — a testing framework designed specifically for non-deterministic AI output.

Define tests in YAML:

name: "billing-agent-tests"
tests:
  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"

  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5

Run them:

npm install -g @ozperium/agentspec
agentspec init
agentspec run

Output:

AgentSpec v1.2.0

  ✓ routes billing question (312ms)
  ✓ handles expired token (891ms)
  ✗ response is on-topic
    Expected semantic similarity ≥ 0.5, got 0.31
    Input: "how do I reset my password?"
    Output: "Please contact support for account issues."

2 passed, 1 failed

Standard test assertions assume deterministic output. AI output isn't.

AgentSpec provides:

contains

/ not_contains

contains_any

/ contains_all

semantically_similar

regex

tool_called

max_latency_ms

llm_judge

agentspec run --ci   # exits 1 on failure, JUnit XML output

Drop it in GitHub Actions:

- name: Run AgentSpec
  run: agentspec run --ci

Now every PR that changes a prompt, model, or tool gets behavior regression tests.

Run with --diff

to compare against a previous baseline:

agentspec run --diff baseline.json

Shows exactly what changed between runs — useful when you swap models or update prompts.

Point it at any running agent endpoint:

agentspec run --endpoint http://localhost:3000/chat

No SDK integration required.

npm install -g @ozperium/agentspec
agentspec init
agentspec run

GitHub: https://github.com/Ozperium/agentspec

Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.