Regression testing for AI agents. Define what your agent should do in plain YAML, run it against your actual agent (any CLI command or HTTP endpoint), and let an LLM judge score every response against your stated criteria — pass/fail, with a reason. Wire it into CI so a prompt change, a tool swap, or a model upgrade can't silently break behavior your users depend on.
This is deliberately narrow: it is not a production-observability platform (that's
Langfuse / Braintrust / Arize territory, and they're well-funded — don't compete head-on).
It's the thing almost nobody has built well yet: a fast, dev-friendly pre-deployment
check that fits in a GitHub Actions step the same way pytest
does.
- Full observability/eval platforms have raised $50–80M rounds in the last year and are actively crowding the "log + monitor production agent traffic" space.
- Almost none of them are built as a lightweight, git-native regression suite a solo developer can add to CI in five minutes — that gap is the wedge.
- Distribution is self-serve/PLG (open-source CLI, developer audience) rather than enterprise sales — the thing solo AI founders are consistently worst at.
pip install -e .
export ANTHROPIC_API_KEY=sk-...
agentcheck run examples/tests.yaml
See examples/tests.yaml
. Each test case specifies an input, a plain-English description of what a correct response looks like, and one of two ways to reach your agent:
command: "..."
— run it as a subprocess;input
is piped to stdin, stdout is captured as the output. Works with any language.agent: "module.path:function_name"
— import that module and call the function in-process withinput
as its only argument; its return value is the output. Useful for LangGraph/CrewAI/Claude-Agent-SDK-style agents that are Python callables rather than standalone CLI scripts — seeexamples/inprocess_agent.py
.
Exactly one of the two is required per case. Either way, the output is scored pass/fail with a one-line reason by an LLM judge — no brittle string matching.
agentcheck run tests.yaml --json-out results.json
writes a JSON report you can
upload as a build artifact (see examples/.github/workflows/agentcheck.yml
).
Add --post-pr-comment
and, on a pull-request run with GITHUB_TOKEN
set (the job
needs permissions: pull-requests: write
), agentcheck posts a markdown summary table as a PR comment, updating the same comment on repeat runs instead of piling up new ones. It's a silent no-op everywhere else (pushes, local runs), so it's safe to leave on in every CI invocation.
A flat pass count ("18/20 passed") doesn't tell you whether a change helped or hurt —
you have to go read the table. --baseline
fixes that by diffing the current run
against a previous --json-out
report, keyed by test name:
agentcheck run tests.yaml --json-out results.json --baseline baseline.json
Try it locally against the bundled example:
agentcheck run examples/tests.yaml --baseline examples/baseline.json
Every test lands in one bucket: unchanged (same pass/fail as the baseline), regressed (baseline passed, now fails — this is the one you care about), improved (baseline failed, now passes), new (not in the baseline), or removed (in the baseline but not in this run — probably a deleted test case, worth a glance). The console prints a one-line summary plus a table of regressions and improvements; a missing or unreadable baseline (there's no baseline yet on a repo's first run) prints a warning and falls back to the plain pass/fail report instead of failing the whole run.
--post-pr-comment
picks this up automatically when --baseline
is also set, so the PR comment leads with "vs baseline: 2 unchanged, 1 improved, 1 regressed" and calls out the regressions specifically, instead of just restating the full results table.
To actually wire this into CI you need somewhere for the baseline to come from — the
usual pattern is: on every push to your default branch, run agentcheck with
--json-out results/baseline.json
and commit that file back to the repo; on every PR,
read the base branch's copy of that file (git show origin/main:results/baseline.json
)
and pass it as --baseline
. See examples/.github/workflows/agentcheck.yml
for a full working version of that.
pip install -e ".[dev]"
pytest
This week: get this CLI working end to end against your own toy agent. Dogfood it.Week 2: open-source it. Post it where agent builders actually hang out (r/LocalLLaMA, the LangChain/LlamaIndex Discords, Hacker News "Show HN", relevant X threads). The goal isn't virality — it's finding 10-20 people who hit this exact pain point and will tell you what's missing.Week 3-4: add the thing they ask for most. Likely candidates: a hosted dashboard for run history, Slack/GitHub PR-comment reporting, or support for a specific popular agent framework (LangGraph, CrewAI, the Claude Agent SDK) as a first-class integration.Only once people are using the free CLI regularly: introduce a paid hosted tier (run history, team sharing, trend charts) — don't build monetization before you have free users who'd miss it if it disappeared.
Don't try to become a general observability platform — that's the crowded, well-funded lane. Stay the "fast, git-native regression check" tool. Depth in one narrow job beats breadth against funded competitors.
MIT — see LICENSE
.