Show HN: Agent-roast – Catches coding agents that weaken tests to get CI green

A developer tool called agent-roast, released as an open-source project under the Nari-Ab GitHub organization, detects when coding agents make a failing CI build pass by weakening tests rather than fixing code, flagging changes such as `toBe(42)` becoming `toBeDefined()`, lowered coverage thresholds, and skipped tests. agent-roast runs deterministically and offline with no model, tokens, or network, and installs as an agent hook for Claude Code, Codex and Cursor, a Git hook, a pull request check via the `Nari-Ab/agent-roast@v1` GitHub Action, or a read-only MCP tool invoked with `agent-roast mcp`. When a check is weakened, the agent is sent back once with the file, line and before/after diff; the project's author notes the agent hooks are tested with simulated hook input and not yet in long live sessions with every agent.

agent-roast catches the changes that turn a red build green by weakening the checks instead of fixing the code : a loosened assertion, a skipped or deleted test, a lowered coverage threshold, a CI step allowed to fail. A linter cannot see these, because every new line looks fine on its own. agent-roast compares the before and after of each change, as an agent hook, a Git hook and a pull request check. It is deterministic and offline: no model, no tokens, no network. <sub Real output of npx agent-roast demo , shortened. The scenario is scripted; the review is not.</sub npx agent-roast demo ten seconds: a scripted example in a temporary repository, nothing of yours is read npx agent-roast review what you have not committed yet on a clean tree: replay your last commits npx agent-roast hook install Claude Code, Codex and Cursor For Claude Code there is also a plugin: /plugin install agent-roast --marketplace Nari-Ab/agent-roast . Any agent that speaks MCP can also call agent-roast as a read-only tool, agent-roast mcp setup https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md mcp-server . When the agent is about to finish, agent-roast reviews everything it changed. If a check got weaker, the agent is sent back once , with the file, the line and the before and after, and told to fix the code instead. If it tries again, you get a warning rather than an endless loop. Anything agent-roast cannot check lets the agent finish: it never breaks a session. An intended change is accepted by committing it. Not sure yet? npx agent-roast hook install --advisory only warns you and never sends the agent back. Setup for each agent, what it does not do, and how to pause it: docs/integrations.md https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md . The agent hooks follow each agent's documented hook protocol and are tested with simulated hook input, not yet in long live sessions with every agent. Reports of what you see are welcome. .github/workflows/agent-roast.yml name: Agent Roast on: pull request permissions: { contents: read, pull-requests: write } jobs: audit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: { fetch-depth: 0 } - uses: Nari-Ab/agent-roast@v1 with: { comment: true } one comment, updated on every push; add advisory: true to try it without failing PRs Also: npx agent-roast review --base origin/main in any CI, npx agent-roast init --hook or the pre-commit framework https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md the-pre-commit-framework for the next commit, and SARIF output for code scanning. Policy is read from the base branch, so a pull request cannot switch off the rules that review it. Before you decide, npx agent-roast replay shows what it would have said about your last 50 commits. | Rule | Fires when | By default | |---|---|---| | assertion-weakened | an exact check is replaced by a weaker one: toBe 42 → toBeDefined , toBe 2 → toBeLessThanOrEqual 2 , toEqual x → toMatchObject x , assert x == 1 → assert x | blocks | | guard-weakened | a coverage threshold is lowered, strict is switched off, a lint rule is downgraded, a CI step may fail, a workflow stops running tests, a conftest.py hook sets a report to passed | blocks | | test-skip | a test that was already there is switched off or expected to fail: it.skip , @pytest.mark.skip , @pytest.mark.xfail , it.fails , test.fixme , t.Skip , ignore , @Disabled , Ignore , xit , ... a new test that is skipped is not reported | blocks | | test-focus , trivial-assert | .only / fit ; expect true .toBe true and its relatives in ten languages; a test that asserts on a function it mocked itself | blocks | | assertion-removed , test-removed , expectation-rewritten | a test loses assertions, disappears, or its expected value is edited next to production code | warns | | swallowed-error , type-escape , stub-abandonment | catch {} / except: pass , as any / type: ignore / pragma: no cover / istanbul ignore , throw new Error "Not implemented" | warns | Renames, moved tests, parametrization, moved assertions and reformatting are recognized and not reported, and neither is a skip that depends on the environment if testing.Short { t.Skip } , test.skip isMobile, ... . JavaScript/TypeScript, Python and Go get the full before/after comparison; skips and placeholder assertions also cover Rust, Java, Kotlin, C , Ruby, PHP and Swift; CI and config files are checked whatever the language. Each rule with examples and limits: docs/rules.md https://github.com/Nari-Ab/agent-roast/blob/main/docs/rules.md . Rules, severities and paths live in .agentroastrc.json at the repository root npx agent-roast init writes a starting file : { "failOn": "assertion-weakened", "guard-weakened", "test-skip", "test-focus", "trivial-assert" , "rules": { "type-escape": "off" }, "ignorePaths": " /generated/ " } A change cannot edit this file to relax its own review, because it is read from HEAD or the base branch. To accept one intentional finding, add an expiring exception instead of turning a rule off, and adopt an existing project without blocking on old debt with a baseline: docs/reference.md https://github.com/Nari-Ab/agent-roast/blob/main/docs/reference.md configuration . Measured, with the denominators and the caveats in docs/benchmarks.md https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md . Weakened checks are rare, mostly deliberate, and not specific to AI. agent-roast is a tripwire that asks about them. On repositories it had not seen, between a quarter and two thirds of its blocks were questions with a harmless answer, depending on the sample. - 145 of 145 hand-labeled cheating edits are caught and 127 of 127 harmless look-alikes reformats, moves, raised thresholds, environment-gated skips stay silent. The cases were written with the detector, so they guard against regressions; they do not prove accuracy. One gap is documented and still open. - 12,645 commits that touch tests or CI/config in 166 public repositories: blocking rules fired on 0.3% of AI-signed commits Claude, Copilot, Cursor, Codex, Devin, aider and on 0.3% of the others. Inside the same repositories the odds for an AI-signed commit were 0.81 times those of another commit 95% CI 0.40–1.67 . There is no sign that AI-signed commits weaken checks more often. - On repositories it was not written from, between a quarter and two thirds of the blocked commits were false alarms. Six samples of 39 or 40 public repositories 238 in all, 67,843 commits were replayed and every blocked commit was read. The share was 69% for 0.17.0 on the first sample and on the fourth, 51% for 0.18.0 on the fourth and 27% on the fifth, which it had not seen either. This version, frozen before the sixth sample was drawn, blocked 100 of its 11,335 commits: 34 changed a check a reviewer should see an assertion that got weaker, a gate that was relaxed, a test switched off or left focused and 66 were false alarms, but 51 of the 66 were one defect in one repository, an array of mypy overrides read as one table, which is fixed in this release; the released build blocks 44 of those commits, 34 and 10. A new test that asserts nothing counts as a block for a reviewer; counted as false alarms, 13 of 33 on the fifth sample and 72 of 100 on the sixth. Most of the rest, in the earlier measurements too, were deliberate and explained. One reader an AI assistant sorted them, and the fixes were made from the same sorting, so treat the figures as a first estimate section 9 https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md 9-repositories-nobody-had-tuned-on , section 10 https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md 10-two-more-samples-and-what-the-fixes-from-sample-4-did . - 9,732 commits compared in full from 164 public repositories that use coding agents 5,454 of them AI-signed were reviewed again for the last two releases, and every new finding was read. 0.15.0's rules reported 30 new findings there and 7 were false alarms fixed in 0.16.0 ; 0.17.0's rule for an exact check turned into a bound reported 18 in 14 commits, all of them loosened checks, after two false alarms were fixed. One reader, and a convenience sample section 7 https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md 7-ai-signed-commits-and-a-test-that-checks-its-own-mock , section 8 https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md 8-an-exact-check-turned-into-a-bound . - replay ran on 179 public repositories that use coding agents: no commit would have been blocked in 149 of them. - The rules for eight more languages Go, Rust, Java, Kotlin, C , Ruby, PHP, Swift ran on the newest 300 commits of 39 repositories and every finding was read: 26 of 99 were skips that depend on the environment, now fixed; of the 73 left, 56 are tests that really are switched off and 17 are deliberate test data. One reader, and the fixes were made on the same repositories, so expect a few new false alarms elsewhere. TDD Guard and Probity enforce a process tests first through agent hooks; aislop scans for broad "AI slop" across 11 languages; hallucinot, fakegreen and SlopGate look at diffs for the same shortcuts; linters and coverage gates see the current code but not that a check got weaker. agent-roast narrows to one question, reads the full before and after, and says how it was measured. Most of these tools are complementary. docs/comparison.md https://github.com/Nari-Ab/agent-roast/blob/main/docs/comparison.md has the details. - Offline. No network request, no model call, no telemetry; a test inspects the bundle for network modules. - Reads, never runs. It reads Git objects or regular files inside the repository for --worktree and never executes project code, hooks, diff programs or package scripts. Reports escape control characters. - Policy from the trusted side. The rules come from HEAD or the base branch, not from the change under review. - Honest about gaps. An incomplete scan exits 2, never "clean". SECURITY.md https://github.com/Nari-Ab/agent-roast/blob/main/SECURITY.md says how to report a problem. It is a review aid. It cannot prove who wrote a change or that code is correct, and it reads text, not program meaning. Rules https://github.com/Nari-Ab/agent-roast/blob/main/docs/rules.md · Integrations https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md · Reference https://github.com/Nari-Ab/agent-roast/blob/main/docs/reference.md commands, exit codes, configuration, exceptions, baselines, limits · Comparison https://github.com/Nari-Ab/agent-roast/blob/main/docs/comparison.md · Measurements https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md · Try it on your repository https://github.com/Nari-Ab/agent-roast/blob/main/docs/pilot.md · Contributing https://github.com/Nari-Ab/agent-roast/blob/main/CONTRIBUTING.md · Changelog https://github.com/Nari-Ab/agent-roast/blob/main/CHANGELOG.md Found a finding that was wrong, or a shortcut it missed? That is the most useful report there is: open an issue https://github.com/Nari-Ab/agent-roast/issues/new/choose . MIT https://github.com/Nari-Ab/agent-roast/blob/main/LICENSE . Created by Nariman Abdukarimov.