{"slug": "testing-ai-agents-is-hard-i-built-a-framework-for-it", "title": "Testing AI agents is hard. I built a framework for it.", "summary": "A developer built AgentSpec, an open-source testing framework for AI agents that handles non-deterministic output. The framework allows defining tests in YAML with assertions like semantic similarity, tool calls, and latency checks, and integrates with CI pipelines via GitHub Actions.", "body_md": "Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly.\n\nYou don't find out until a user complains.\n\nThe problem: AI agents are non-deterministic. Traditional unit tests don't work. `expect(output).toBe(\"transfer to billing\")`\n\nfails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.\n\nI built [AgentSpec](https://github.com/Ozperium/agentspec) to fix this — a testing framework designed specifically for non-deterministic AI output.\n\nDefine tests in YAML:\n\n```\nname: \"billing-agent-tests\"\ntests:\n  - name: \"routes billing question\"\n    input: \"I want a refund\"\n    expect:\n      contains_any: [\"billing\", \"refund\", \"support\"]\n      tool_called: \"transfer_to_billing\"\n\n  - name: \"handles expired token\"\n    input: \"my token expired\"\n    expect:\n      contains: \"refresh token\"\n      not_contains: \"I don't know\"\n      max_latency_ms: 5000\n\n  - name: \"response is on-topic\"\n    input: \"how do I reset my password?\"\n    expect:\n      semantically_similar: \"reset password credentials\"\n      min_confidence: 0.5\n```\n\nRun them:\n\n```\nnpm install -g @ozperium/agentspec\nagentspec init\nagentspec run\n```\n\nOutput:\n\n```\nAgentSpec v1.2.0\n\n  ✓ routes billing question (312ms)\n  ✓ handles expired token (891ms)\n  ✗ response is on-topic\n    Expected semantic similarity ≥ 0.5, got 0.31\n    Input: \"how do I reset my password?\"\n    Output: \"Please contact support for account issues.\"\n\n2 passed, 1 failed\n```\n\nStandard test assertions assume deterministic output. AI output isn't.\n\nAgentSpec provides:\n\n`contains`\n\n/ `not_contains`\n\n`contains_any`\n\n/ `contains_all`\n\n`semantically_similar`\n\n`regex`\n\n`tool_called`\n\n`max_latency_ms`\n\n`llm_judge`\n\n```\nagentspec run --ci   # exits 1 on failure, JUnit XML output\n```\n\nDrop it in GitHub Actions:\n\n```\n- name: Run AgentSpec\n  run: agentspec run --ci\n```\n\nNow every PR that changes a prompt, model, or tool gets behavior regression tests.\n\nRun with `--diff`\n\nto compare against a previous baseline:\n\n```\nagentspec run --diff baseline.json\n```\n\nShows exactly what changed between runs — useful when you swap models or update prompts.\n\nPoint it at any running agent endpoint:\n\n```\nagentspec run --endpoint http://localhost:3000/chat\n```\n\nNo SDK integration required.\n\n```\nnpm install -g @ozperium/agentspec\nagentspec init\nagentspec run\n```\n\nGitHub: [https://github.com/Ozperium/agentspec](https://github.com/Ozperium/agentspec)\n\n*Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.*", "url": "https://wpnews.pro/news/testing-ai-agents-is-hard-i-built-a-framework-for-it", "canonical_source": "https://dev.to/pawfromoz/testing-ai-agents-is-hard-i-built-a-framework-for-it-3hk0", "published_at": "2026-07-24 04:22:54+00:00", "updated_at": "2026-07-24 05:01:34.496013+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-infrastructure"], "entities": ["AgentSpec", "Ozperium", "GitHub Actions", "AICostTracker"], "alternates": {"html": "https://wpnews.pro/news/testing-ai-agents-is-hard-i-built-a-framework-for-it", "markdown": "https://wpnews.pro/news/testing-ai-agents-is-hard-i-built-a-framework-for-it.md", "text": "https://wpnews.pro/news/testing-ai-agents-is-hard-i-built-a-framework-for-it.txt", "jsonld": "https://wpnews.pro/news/testing-ai-agents-is-hard-i-built-a-framework-for-it.jsonld"}}