# AgentCheck – regression testing for AI agents, with diff-aware CI reports > Source: > Published: 2026-08-21 20:21:15+00:00 **Regression testing for AI agents.** Define what your agent should do in plain YAML, run it against your actual agent (any CLI command or HTTP endpoint), and let an LLM judge score every response against your stated criteria — pass/fail, with a reason. Wire it into CI so a prompt change, a tool swap, or a model upgrade can't silently break behavior your users depend on. This is deliberately narrow: it is not a production-observability platform (that's Langfuse / Braintrust / Arize territory, and they're well-funded — don't compete head-on). It's the thing almost nobody has built well yet: a fast, dev-friendly **pre-deployment check** that fits in a GitHub Actions step the same way `pytest` does. - Full observability/eval platforms have raised $50–80M rounds in the last year and are actively crowding the "log + monitor production agent traffic" space. - Almost none of them are built as a lightweight, git-native regression suite a solo developer can add to CI in five minutes — that gap is the wedge. - Distribution is self-serve/PLG (open-source CLI, developer audience) rather than enterprise sales — the thing solo AI founders are consistently worst at. ``` pip install -e . export ANTHROPIC_API_KEY=sk-... agentcheck run examples/tests.yaml ``` See `examples/tests.yaml` . Each test case specifies an input, a plain-English description of what a correct response looks like, and one of two ways to reach your agent: `command: "..."` — run it as a subprocess;`input` is piped to stdin, stdout is captured as the output. Works with any language.`agent: "module.path:function_name"` — import that module and call the function in-process with`input` as its only argument; its return value is the output. Useful for LangGraph/CrewAI/Claude-Agent-SDK-style agents that are Python callables rather than standalone CLI scripts — see`examples/inprocess_agent.py` . Exactly one of the two is required per case. Either way, the output is scored pass/fail with a one-line reason by an LLM judge — no brittle string matching. `agentcheck run tests.yaml --json-out results.json` writes a JSON report you can upload as a build artifact (see `examples/.github/workflows/agentcheck.yml` ). Add `--post-pr-comment` and, on a pull-request run with `GITHUB_TOKEN` set (the job needs `permissions: pull-requests: write` ), agentcheck posts a markdown summary table as a PR comment, updating the same comment on repeat runs instead of piling up new ones. It's a silent no-op everywhere else (pushes, local runs), so it's safe to leave on in every CI invocation. A flat pass count ("18/20 passed") doesn't tell you whether a change helped or hurt — you have to go read the table. `--baseline` fixes that by diffing the current run against a previous `--json-out` report, keyed by test name: ``` agentcheck run tests.yaml --json-out results.json --baseline baseline.json ``` Try it locally against the bundled example: ``` agentcheck run examples/tests.yaml --baseline examples/baseline.json ``` Every test lands in one bucket: **unchanged** (same pass/fail as the baseline), **regressed** (baseline passed, now fails — this is the one you care about), **improved** (baseline failed, now passes), **new** (not in the baseline), or **removed** (in the baseline but not in this run — probably a deleted test case, worth a glance). The console prints a one-line summary plus a table of regressions and improvements; a missing or unreadable baseline (there's no baseline yet on a repo's first run) prints a warning and falls back to the plain pass/fail report instead of failing the whole run. `--post-pr-comment` picks this up automatically when `--baseline` is also set, so the PR comment leads with "vs baseline: 2 unchanged, 1 improved, 1 regressed" and calls out the regressions specifically, instead of just restating the full results table. To actually wire this into CI you need somewhere for the baseline to come from — the usual pattern is: on every push to your default branch, run agentcheck with `--json-out results/baseline.json` and commit that file back to the repo; on every PR, read the base branch's copy of that file (`git show origin/main:results/baseline.json` ) and pass it as `--baseline` . See `examples/.github/workflows/agentcheck.yml` for a full working version of that. ``` pip install -e ".[dev]" pytest ``` **This week:** get this CLI working end to end against your own toy agent. Dogfood it.**Week 2:** open-source it. Post it where agent builders actually hang out (r/LocalLLaMA, the LangChain/LlamaIndex Discords, Hacker News "Show HN", relevant X threads). The goal isn't virality — it's finding 10-20 people who hit this exact pain point and will tell you what's missing.**Week 3-4:** add the thing they ask for most. Likely candidates: a hosted dashboard for run history, Slack/GitHub PR-comment reporting, or support for a specific popular agent framework (LangGraph, CrewAI, the Claude Agent SDK) as a first-class integration.**Only once people are using the free CLI regularly:** introduce a paid hosted tier (run history, team sharing, trend charts) — don't build monetization before you have free users who'd miss it if it disappeared. Don't try to become a general observability platform — that's the crowded, well-funded lane. Stay the "fast, git-native regression check" tool. Depth in one narrow job beats breadth against funded competitors. MIT — see `LICENSE` .