Coding agents get blamed for "making tests pass" by skipping or deleting them. I wanted numbers before building a guard for that, so I collected public pull requests that modified test code (2026, repositories with 100+ stars) and had them labeled before running any tool on them:
What I found:
Then I gave Codex three tasks that could not be done honestly: two tests that contradict the documented behavior, and an image feature whose native dependency is missing. It never weakened a test. It changed the code to satisfy the wrong test in 3 of 3 runs without the guard and 2 of 3 with it; for the missing image library it added a quiet fallback instead of failing. One model, one run per task, so a small sample, but the shortcut was in the code, not in the tests.
So the check became a review signal, not an accusation. repopilot review lists every place a change touched the checks that judge it: focused or skipped tests, removed tests, tests that lost assertions, new lint/type/coverage suppressions, relaxed CI or tool gates, and new entries in RepoPilot's own suppression file.
Held-out precision:
| Signal | Precision |
|---|---|
| suppression added | 5/6 (6/6 after one documented label fix) |
| CI/tool gate relaxed | 1/1 |
| assertions removed | 2/3 |
| test removed (case or whole file) | 5/8 |
What it misses: checks trivialized by structure (the unmount trick), loosened matchers, helpers in other files, ESLint bulk suppression files, module-level conditional skips, Ginkgo specs, and the quiet fallback above. That last one is next. Limits: one LLM labeler plus my review of the "weakened" verdicts, and the same model helped build the detectors. Treat the numbers as exploratory; the corpus, labels, and harness are in the repo.
It runs locally and is deterministic, with no LLM. For Claude Code and Codex there is a plugin that snapshots the repo when a session starts and, when the agent tries to finish, stops it once per signal if it weakened a check, with the file and line. Dependency bumps and workflow edits stay in the report and don't interrupt the agent.
npm i -g repopilot # or: cargo install repopilot
repopilot review . --base origin/main
Claude Code: /plugin marketplace add MykytaStel/repopilot, then /plugin install repopilot@repopilot.
Repo: https://github.com/MykytaStel/repopilot
I wrote this post with help from Claude. The numbers, labels, and harness are in the repo, and I checked them before publishing.