{"slug": "show-hn-agent-roast-catches-coding-agents-that-weaken-tests-to-get-ci-green", "title": "Show HN: Agent-roast – Catches coding agents that weaken tests to get CI green", "summary": "A developer tool called agent-roast, released as an open-source project under the Nari-Ab GitHub organization, detects when coding agents make a failing CI build pass by weakening tests rather than fixing code, flagging changes such as `toBe(42)` becoming `toBeDefined()`, lowered coverage thresholds, and skipped tests. agent-roast runs deterministically and offline with no model, tokens, or network, and installs as an agent hook for Claude Code, Codex and Cursor, a Git hook, a pull request check via the `Nari-Ab/agent-roast@v1` GitHub Action, or a read-only MCP tool invoked with `agent-roast mcp`. When a check is weakened, the agent is sent back once with the file, line and before/after diff; the project's author notes the agent hooks are tested with simulated hook input and not yet in long live sessions with every agent.", "body_md": "**agent-roast** catches the changes that turn a red build green by weakening the *checks* instead of fixing the *code*: a\nloosened assertion, a skipped or deleted test, a lowered coverage threshold, a CI step allowed to fail. A linter cannot see\nthese, because every new line looks fine on its own. agent-roast compares the **before and after** of each change, as an\nagent hook, a Git hook and a pull request check. It is deterministic and offline: no model, no tokens, no network.\n\n<sub>Real output of `npx agent-roast demo`, shortened. The scenario is scripted; the review is not.</sub>\n\n```\nnpx agent-roast demo     # ten seconds: a scripted example in a temporary repository, nothing of yours is read\nnpx agent-roast          # review what you have not committed yet (on a clean tree: replay your last commits)\nnpx agent-roast hook install          # Claude Code, Codex and Cursor\n```\n\nFor Claude Code there is also a plugin: `/plugin install agent-roast --marketplace Nari-Ab/agent-roast`. Any agent that speaks\nMCP can also call agent-roast as a read-only tool, `agent-roast mcp` ([setup](https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md#mcp-server)).\n\nWhen the agent is about to finish, agent-roast reviews everything it changed. If a check got weaker, the agent is **sent back\nonce**, with the file, the line and the before and after, and told to fix the code instead. If it tries again, you get a\nwarning rather than an endless loop. Anything agent-roast cannot check lets the agent finish: it never breaks a session. An\nintended change is accepted by committing it. Not sure yet? `npx agent-roast hook install --advisory` only warns you and never\nsends the agent back. Setup for each agent, what it does not do, and how to pause it: [docs/integrations.md](https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md).\n\nThe agent hooks follow each agent's documented hook protocol and are tested with simulated hook input, not yet in long live sessions with every agent. Reports of what you see are welcome.\n\n```\n# .github/workflows/agent-roast.yml\nname: Agent Roast\non: pull_request\npermissions: { contents: read, pull-requests: write }\njobs:\n  audit:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n        with: { fetch-depth: 0 }\n      - uses: Nari-Ab/agent-roast@v1\n        with: { comment: true }      # one comment, updated on every push; add `advisory: true` to try it without failing PRs\n```\n\nAlso: `npx agent-roast review --base origin/main` in any CI, `npx agent-roast init --hook` or the\n[pre-commit framework](https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md#the-pre-commit-framework) for the next commit, and SARIF output for code scanning.\nPolicy is read from the **base** branch, so a pull request cannot switch off the rules that review it. Before you decide,\n`npx agent-roast replay` shows what it would have said about your last 50 commits.\n\n| Rule | Fires when | By default | \n|---|---|---|\n| `assertion-weakened` | an exact check is replaced by a weaker one: `toBe(42)` →`toBeDefined()` ,`toBe(2)` →`toBeLessThanOrEqual(2)` ,`toEqual(x)` →`toMatchObject(x)` ,`assert x == 1` →`assert x` | blocks | \n| `guard-weakened` | a coverage threshold is lowered, `strict` is switched off, a lint rule is downgraded, a CI step may fail, a workflow stops running tests, a`conftest.py` hook sets a report to`passed` | blocks | \n| `test-skip` | a test that was already there is switched off or expected to fail: `it.skip` ,`@pytest.mark.skip` ,`@pytest.mark.xfail` ,`it.fails` ,`test.fixme` ,`t.Skip` ,`#[ignore]` ,`@Disabled` ,`[Ignore]` ,`xit` , ... (a new test that is skipped is not reported) | blocks | \n| `test-focus` ,`trivial-assert` | `.only` /`fit` ;`expect(true).toBe(true)` and its relatives in ten languages; a test that asserts on a function it mocked itself | blocks | \n| `assertion-removed` ,`test-removed` ,`expectation-rewritten` | a test loses assertions, disappears, or its expected value is edited next to production code | warns | \n| `swallowed-error` ,`type-escape` ,`stub-abandonment` | `catch {}` /`except: pass` ,`as any` /`# type: ignore` /`# pragma: no cover` /`istanbul ignore` ,`throw new Error(\"Not implemented\")` | warns | \n\nRenames, moved tests, parametrization, moved assertions and reformatting are recognized and not reported, and neither is a skip\nthat depends on the environment (`if testing.Short() { t.Skip() }`, `test.skip(isMobile, ...)`). JavaScript/TypeScript, Python\nand Go get the full before/after comparison; skips and placeholder assertions also cover Rust, Java, Kotlin, C#, Ruby, PHP and\nSwift; CI and config files are checked whatever the language. Each rule with examples and limits: [docs/rules.md](https://github.com/Nari-Ab/agent-roast/blob/main/docs/rules.md).\n\nRules, severities and paths live in `.agentroastrc.json` at the repository root (`npx agent-roast init` writes a starting file):\n\n```\n{\n  \"failOn\": [\"assertion-weakened\", \"guard-weakened\", \"test-skip\", \"test-focus\", \"trivial-assert\"],\n  \"rules\": { \"type-escape\": \"off\" },\n  \"ignorePaths\": [\"**/generated/**\"]\n}\n```\n\nA change cannot edit this file to relax its own review, because it is read from `HEAD` or the base branch. To accept one\nintentional finding, add an expiring exception instead of turning a rule off, and adopt an existing project without blocking on\nold debt with a baseline: [docs/reference.md](https://github.com/Nari-Ab/agent-roast/blob/main/docs/reference.md#configuration).\n\nMeasured, with the denominators and the caveats in [docs/benchmarks.md](https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md). **Weakened checks are rare, mostly\ndeliberate, and not specific to AI. agent-roast is a tripwire that asks about them. On repositories it had not seen, between a quarter and\ntwo thirds of its blocks were questions with a harmless answer, depending on the sample.**\n\n- **145 of 145** hand-labeled cheating edits are caught and**127 of 127** harmless look-alikes (reformats, moves, raised\nthresholds, environment-gated skips) stay silent. The cases were written with the detector, so they guard against regressions;\nthey do not prove accuracy. One gap is documented and still open.\n- **12,645 commits** that touch tests or CI/config in 166 public repositories: blocking rules fired on 0.3% of AI-signed commits\n(Claude, Copilot, Cursor, Codex, Devin, aider) and on 0.3% of the others. Inside the same repositories the odds for an\nAI-signed commit were 0.81 times those of another commit (95% CI 0.40–1.67). There is no sign that AI-signed commits weaken\nchecks more often.\n- **On repositories it was not written from, between a quarter and two thirds of the blocked commits were false alarms.** Six samples of 39\nor 40 public repositories (238 in all, 67,843 commits) were replayed and every blocked commit was read. The share was 69% for 0.17.0 on the\nfirst sample and on the fourth, 51% for 0.18.0 on the fourth and 27% on the fifth, which it had not seen either. This version, frozen before the\nsixth sample was drawn, blocked 100 of its 11,335 commits: 34 changed a check a reviewer should see (an assertion that got weaker, a gate\nthat was relaxed, a test switched off or left focused) and 66 were false alarms, but 51 of the 66 were one defect in one repository, an array of\nmypy overrides read as one table, which is fixed in this release; the released build blocks 44 of those commits, 34 and 10. A new\ntest that asserts nothing counts as a block for a reviewer; counted as false alarms, 13 of 33 on the fifth sample and 72 of 100 on the sixth. Most of the rest, in the\nearlier measurements too, were deliberate and explained. One reader (an AI assistant) sorted them, and the fixes were made from the same\nsorting, so treat the figures as a first estimate ([section 9](https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md#9-repositories-nobody-had-tuned-on) ,[section 10](https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md#10-two-more-samples-and-what-the-fixes-from-sample-4-did) ).\n- **9,732 commits compared in full** from 164 public repositories that use coding agents (5,454 of them AI-signed) were reviewed again for the\nlast two releases, and every new finding was read. 0.15.0's rules reported 30 new findings there and 7 were false alarms (fixed in 0.16.0);\n0.17.0's rule for an exact check turned into a bound reported 18 in 14 commits, all of them loosened checks, after two false alarms were\nfixed. One reader, and a convenience sample ([section 7](https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md#7-ai-signed-commits-and-a-test-that-checks-its-own-mock) ,[section 8](https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md#8-an-exact-check-turned-into-a-bound) ).\n- `replay` ran on 179 public repositories that use coding agents: no commit would have been blocked in 149 of them.\n- The rules for **eight more languages** (Go, Rust, Java, Kotlin, C#, Ruby, PHP, Swift) ran on the newest 300 commits of 39\nrepositories and every finding was read: 26 of 99 were skips that depend on the environment, now fixed; of the 73 left, 56 are\ntests that really are switched off and 17 are deliberate test data. One reader, and the fixes were made on the same\nrepositories, so expect a few new false alarms elsewhere.\n\nTDD Guard and Probity enforce a *process* (tests first) through agent hooks; aislop scans for broad \"AI slop\" across 11\nlanguages; hallucinot, fakegreen and SlopGate look at diffs for the same shortcuts; linters and coverage gates see the current\ncode but not that a check got weaker. agent-roast narrows to one question, reads the full before and after, and says how it was\nmeasured. Most of these tools are complementary. [docs/comparison.md](https://github.com/Nari-Ab/agent-roast/blob/main/docs/comparison.md) has the details.\n\n- **Offline.** No network request, no model call, no telemetry; a test inspects the bundle for network modules.\n- **Reads, never runs.** It reads Git objects (or regular files inside the repository for`--worktree` ) and never executes\nproject code, hooks, diff programs or package scripts. Reports escape control characters.\n- **Policy from the trusted side.** The rules come from`HEAD` or the base branch, not from the change under review.\n- **Honest about gaps.** An incomplete scan exits 2, never \"clean\".[SECURITY.md](https://github.com/Nari-Ab/agent-roast/blob/main/SECURITY.md) says how to report a problem.\n\nIt is a review aid. It cannot prove who wrote a change or that code is correct, and it reads text, not program meaning.\n\n[Rules](https://github.com/Nari-Ab/agent-roast/blob/main/docs/rules.md) · [Integrations](https://github.com/Nari-Ab/agent-roast/blob/main/docs/integrations.md) · [Reference](https://github.com/Nari-Ab/agent-roast/blob/main/docs/reference.md) (commands, exit codes,\nconfiguration, exceptions, baselines, limits) · [Comparison](https://github.com/Nari-Ab/agent-roast/blob/main/docs/comparison.md) · [Measurements](https://github.com/Nari-Ab/agent-roast/blob/main/docs/benchmarks.md) ·\n[Try it on your repository](https://github.com/Nari-Ab/agent-roast/blob/main/docs/pilot.md) · [Contributing](https://github.com/Nari-Ab/agent-roast/blob/main/CONTRIBUTING.md) · [Changelog](https://github.com/Nari-Ab/agent-roast/blob/main/CHANGELOG.md)\n\nFound a finding that was wrong, or a shortcut it missed? That is the most useful report there is:\n[open an issue](https://github.com/Nari-Ab/agent-roast/issues/new/choose).\n\n[MIT](https://github.com/Nari-Ab/agent-roast/blob/main/LICENSE). Created by Nariman Abdukarimov.", "url": "https://wpnews.pro/news/show-hn-agent-roast-catches-coding-agents-that-weaken-tests-to-get-ci-green", "canonical_source": "https://github.com/Nari-Ab/agent-roast/", "published_at": "2026-11-12 15:03:06+00:00", "updated_at": "2026-11-12 15:21:51.957088+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "ai-safety"], "entities": ["agent-roast", "Nari-Ab", "Claude Code", "Codex", "Cursor", "GitHub Actions", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-agent-roast-catches-coding-agents-that-weaken-tests-to-get-ci-green", "markdown": "https://wpnews.pro/news/show-hn-agent-roast-catches-coding-agents-that-weaken-tests-to-get-ci-green.md", "text": "https://wpnews.pro/news/show-hn-agent-roast-catches-coding-agents-that-weaken-tests-to-get-ci-green.txt", "jsonld": "https://wpnews.pro/news/show-hn-agent-roast-catches-coding-agents-that-weaken-tests-to-get-ci-green.jsonld"}}