{"slug": "agentcheck-regression-testing-for-ai-agents-with-diff-aware-ci-reports", "title": "AgentCheck – regression testing for AI agents, with diff-aware CI reports", "summary": "AgentCheck, a new open-source regression testing tool for AI agents, lets developers define expected agent behavior in YAML and run pass/fail checks against any CLI command or HTTP endpoint, with an LLM judge scoring responses. It integrates into CI via GitHub Actions, supports diff-aware reports against a baseline, and can post PR comments summarizing regressions and improvements. The tool targets a gap left by observability platforms like Langfuse, Braintrust, and Arize, which focus on production monitoring rather than lightweight pre-deployment checks.", "body_md": "**Regression testing for AI agents.** Define what your agent should do in plain YAML,\nrun it against your actual agent (any CLI command or HTTP endpoint), and let an LLM judge\nscore every response against your stated criteria — pass/fail, with a reason. Wire it into\nCI so a prompt change, a tool swap, or a model upgrade can't silently break behavior your\nusers depend on.\n\nThis is deliberately narrow: it is not a production-observability platform (that's\nLangfuse / Braintrust / Arize territory, and they're well-funded — don't compete head-on).\nIt's the thing almost nobody has built well yet: a fast, dev-friendly **pre-deployment\ncheck** that fits in a GitHub Actions step the same way `pytest`\n\ndoes.\n\n- Full observability/eval platforms have raised $50–80M rounds in the last year and are actively crowding the \"log + monitor production agent traffic\" space.\n- Almost none of them are built as a lightweight, git-native regression suite a solo developer can add to CI in five minutes — that gap is the wedge.\n- Distribution is self-serve/PLG (open-source CLI, developer audience) rather than enterprise sales — the thing solo AI founders are consistently worst at.\n\n```\npip install -e .\nexport ANTHROPIC_API_KEY=sk-...\nagentcheck run examples/tests.yaml\n```\n\nSee `examples/tests.yaml`\n\n. Each test case specifies an input, a plain-English\ndescription of what a correct response looks like, and one of two ways to reach your\nagent:\n\n`command: \"...\"`\n\n— run it as a subprocess;`input`\n\nis piped to stdin, stdout is captured as the output. Works with any language.`agent: \"module.path:function_name\"`\n\n— import that module and call the function in-process with`input`\n\nas its only argument; its return value is the output. Useful for LangGraph/CrewAI/Claude-Agent-SDK-style agents that are Python callables rather than standalone CLI scripts — see`examples/inprocess_agent.py`\n\n.\n\nExactly one of the two is required per case. Either way, the output is scored pass/fail with a one-line reason by an LLM judge — no brittle string matching.\n\n`agentcheck run tests.yaml --json-out results.json`\n\nwrites a JSON report you can\nupload as a build artifact (see `examples/.github/workflows/agentcheck.yml`\n\n).\n\nAdd `--post-pr-comment`\n\nand, on a pull-request run with `GITHUB_TOKEN`\n\nset (the job\nneeds `permissions: pull-requests: write`\n\n), agentcheck posts a markdown summary table\nas a PR comment, updating the same comment on repeat runs instead of piling up new\nones. It's a silent no-op everywhere else (pushes, local runs), so it's safe to leave\non in every CI invocation.\n\nA flat pass count (\"18/20 passed\") doesn't tell you whether a change helped or hurt —\nyou have to go read the table. `--baseline`\n\nfixes that by diffing the current run\nagainst a previous `--json-out`\n\nreport, keyed by test name:\n\n```\nagentcheck run tests.yaml --json-out results.json --baseline baseline.json\n```\n\nTry it locally against the bundled example:\n\n```\nagentcheck run examples/tests.yaml --baseline examples/baseline.json\n```\n\nEvery test lands in one bucket: **unchanged** (same pass/fail as the baseline),\n**regressed** (baseline passed, now fails — this is the one you care about),\n**improved** (baseline failed, now passes), **new** (not in the baseline), or\n**removed** (in the baseline but not in this run — probably a deleted test case,\nworth a glance). The console prints a one-line summary plus a table of regressions\nand improvements; a missing or unreadable baseline (there's no baseline yet on a\nrepo's first run) prints a warning and falls back to the plain pass/fail report\ninstead of failing the whole run.\n\n`--post-pr-comment`\n\npicks this up automatically when `--baseline`\n\nis also set, so the\nPR comment leads with \"vs baseline: 2 unchanged, 1 improved, 1 regressed\" and calls\nout the regressions specifically, instead of just restating the full results table.\n\nTo actually wire this into CI you need somewhere for the baseline to come from — the\nusual pattern is: on every push to your default branch, run agentcheck with\n`--json-out results/baseline.json`\n\nand commit that file back to the repo; on every PR,\nread the base branch's copy of that file (`git show origin/main:results/baseline.json`\n\n)\nand pass it as `--baseline`\n\n. See `examples/.github/workflows/agentcheck.yml`\n\nfor a full\nworking version of that.\n\n```\npip install -e \".[dev]\"\npytest\n```\n\n**This week:** get this CLI working end to end against your own toy agent. Dogfood it.**Week 2:** open-source it. Post it where agent builders actually hang out (r/LocalLLaMA, the LangChain/LlamaIndex Discords, Hacker News \"Show HN\", relevant X threads). The goal isn't virality — it's finding 10-20 people who hit this exact pain point and will tell you what's missing.**Week 3-4:** add the thing they ask for most. Likely candidates: a hosted dashboard for run history, Slack/GitHub PR-comment reporting, or support for a specific popular agent framework (LangGraph, CrewAI, the Claude Agent SDK) as a first-class integration.**Only once people are using the free CLI regularly:** introduce a paid hosted tier (run history, team sharing, trend charts) — don't build monetization before you have free users who'd miss it if it disappeared.\n\nDon't try to become a general observability platform — that's the crowded, well-funded lane. Stay the \"fast, git-native regression check\" tool. Depth in one narrow job beats breadth against funded competitors.\n\nMIT — see `LICENSE`\n\n.", "url": "https://wpnews.pro/news/agentcheck-regression-testing-for-ai-agents-with-diff-aware-ci-reports", "canonical_source": "https://github.com/rez-99/agentcheck", "published_at": "2026-08-21 20:21:15+00:00", "updated_at": "2026-08-21 20:44:44.904410+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "mlops"], "entities": ["AgentCheck", "Langfuse", "Braintrust", "Arize", "GitHub Actions"], "alternates": {"html": "https://wpnews.pro/news/agentcheck-regression-testing-for-ai-agents-with-diff-aware-ci-reports", "markdown": "https://wpnews.pro/news/agentcheck-regression-testing-for-ai-agents-with-diff-aware-ci-reports.md", "text": "https://wpnews.pro/news/agentcheck-regression-testing-for-ai-agents-with-diff-aware-ci-reports.txt", "jsonld": "https://wpnews.pro/news/agentcheck-regression-testing-for-ai-agents-with-diff-aware-ci-reports.jsonld"}}