Testing AI agents is hard. I built a framework for it.

A developer built AgentSpec, an open-source testing framework for AI agents that handles non-deterministic output. The framework allows defining tests in YAML with assertions like semantic similarity, tool calls, and latency checks, and integrates with CI pipelines via GitHub Actions.

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly. You don't find out until a user complains. The problem: AI agents are non-deterministic. Traditional unit tests don't work. expect output .toBe "transfer to billing" fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways. I built AgentSpec https://github.com/Ozperium/agentspec to fix this — a testing framework designed specifically for non-deterministic AI output. Define tests in YAML: name: "billing-agent-tests" tests: - name: "routes billing question" input: "I want a refund" expect: contains any: "billing", "refund", "support" tool called: "transfer to billing" - name: "handles expired token" input: "my token expired" expect: contains: "refresh token" not contains: "I don't know" max latency ms: 5000 - name: "response is on-topic" input: "how do I reset my password?" expect: semantically similar: "reset password credentials" min confidence: 0.5 Run them: npm install -g @ozperium/agentspec agentspec init agentspec run Output: AgentSpec v1.2.0 ✓ routes billing question 312ms ✓ handles expired token 891ms ✗ response is on-topic Expected semantic similarity ≥ 0.5, got 0.31 Input: "how do I reset my password?" Output: "Please contact support for account issues." 2 passed, 1 failed Standard test assertions assume deterministic output. AI output isn't. AgentSpec provides: contains / not contains contains any / contains all semantically similar regex tool called max latency ms llm judge agentspec run --ci exits 1 on failure, JUnit XML output Drop it in GitHub Actions: - name: Run AgentSpec run: agentspec run --ci Now every PR that changes a prompt, model, or tool gets behavior regression tests. Run with --diff to compare against a previous baseline: agentspec run --diff baseline.json Shows exactly what changed between runs — useful when you swap models or update prompts. Point it at any running agent endpoint: agentspec run --endpoint http://localhost:3000/chat No SDK integration required. npm install -g @ozperium/agentspec agentspec init agentspec run GitHub: https://github.com/Ozperium/agentspec https://github.com/Ozperium/agentspec Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.