What my review gate does about weakened or deleted tests

A developer released two-engines-gate, an MIT-licensed toolset in which Claude Code and Codex review each other's commits, and documented a gap where a commit that weakens tests alongside broken code can still earn a pass record. Testing on 23 October with a small Python project, the developer showed that rewriting two assertions to hold for a deliberately broken add function produced a gate PASS, since the test command exited 0 and tests still ran; catching that case falls to the reviewer prompt, while gate.sh only hard-fails runs where zero tests execute.

If Claude Code weakens a test in the same commit that breaks the code, my test gate passes that commit. Codex is told to look for weakened and deleted tests when it reviews, and the only hard check in the scripts is for a run where zero tests ran. Kartik pointed out this weak spot under my Hashnode post on how Claude Code and Codex review each other's work. I gave him the short answer in the comments. This post is the full answer, with the commands I ran to check it. two-engines-gate https://github.com/IliyaOblakov/two-engines-gate is the free, MIT licensed version of how I make Claude Code and Codex check each other. gate.sh runs your tests and writes a pass record named after the commit, review.sh gets the other engine to review it, and merge-check.sh refuses the merge unless the commit has both. I wrote about the review loop https://rayastats.com/en/blog/how-to-make-codex-review-claude-code-commits?utm source=devto&utm medium=article&utm campaign=kit-3wk-2026-10&utm content=weakened-tests and the commit hash records https://rayastats.com/en/blog/review-verdict-saved-under-commit-hash?utm source=devto&utm medium=article&utm campaign=kit-3wk-2026-10&utm content=weakened-tests before. gate.sh runs whatever tests are in the commit. When a commit changes the code and the tests together, the same commit also decides what a pass means. I checked this on 23 October with the public scripts and a small Python project. It had an add function and two assertEqual tests. First I broke add so it returned a + b + 1 , and the gate failed as it should: FAILED failures=2 gate: FAIL exit 1 on 8390173080fb; no pass record; full log .ai/tests/8390173080fb.log Then, in the next commit, I changed the two tests to assertTrue add 2, 3 and assertIsNotNone add -1, 1 . Both hold for the broken function, so the gate wrote a pass record: gate: running 'python3 -m unittest -q' on 266c4d979b71 gate: PASS on 266c4d979b71, record .ai/tests/266c4d979b7102820377e64ab3f20b069e41bf01.pass The test command exited 0 and two tests still ran, so gate.sh had nothing to refuse. Catching this falls to the reviewer. The free review-prompt.md asks for a static review. Codex reads the commits, the diff and any file in the checkout, and runs no tests or builds. This is the line that covers tests: Check in particular: behaviour the commits promise but the diff does not deliver, error paths and edge cases, tests that would still pass on the old code, and anything that weakens an existing check. The same prompt says a blocker or should-fix needs "a concrete, reproducible failure". For a weakened assertion that failure is easy to write down. Call add 2, 3 on the new code, get 6, and watch the new test pass anyway. A reviewer that reads the test diff next to the code diff has everything it needs to flag it. In my own projects the prompt Codex gets is stricter. Every test change has to fail on the old behaviour, nothing can be weakened without a stated reason, and the reviewer has to check the diff for any weakened or deleted test. A loosened assertion with no reason comes back as CHANGES REQUIRED. The README tells you to edit review-prompt.md to add project rules, so you can put the same three rules in your copy. That is still the reviewer's judgement. The README says so directly: "A review can miss things, and tests only check what they test." If Codex misses the weakened test and answers ACCEPT, merge-check.sh finds a pass record and an ACCEPT for the commit and lets it through. One case is checked by the script itself. gate.sh refuses a pass when the test command prints nothing, or when the test summaries it knows add up to zero tests that ran. It reads the summary lines of pytest, unittest, node --test , mocha, Jest and Vitest. That covers deleting every test, a filter or file list that selects nothing, and, for every runner except unittest, a run where every test was skipped. In my Python project, deleting the test file made unittest exit 5 with "NO TESTS RAN", so the gate already failed on the exit code. To see the zero-test check on its own, I ran the same command with || true so it always exited 0: gate: the test summaries in the output show that no test ran; set GATE COUNT RE or GATE MIN TESTS README gate: FAIL exit 1 on f93de2d8e8a6; no pass record; full log .ai/tests/f93de2d8e8a6.log Deleting one of the two tests is a different case. One test still ran, so the default check passed the commit. If you want the gate to notice a drop in the count, set GATE COUNT RE to a pattern for the count line your runner prints, and GATE MIN TESTS to the number of tests you have now. With GATE COUNT RE='^Ran 0-9 + tests? in ' and GATE MIN TESTS=2 , the same commit failed: gate: the output reports 1 tests, fewer than GATE MIN TESTS=2; set GATE COUNT RE or GATE MIN TESTS README gate: FAIL exit 1 on 1ea182ad01af; no pass record; full log .ai/tests/1ea182ad01af.log The minimum is set where you run the gate, outside the commit, so you raise it by hand when you add tests. go test and cargo test need GATE COUNT RE before any count check works for them, and the README has an example for each. GATE MIN TESTS=0 turns the check off. New tests get one more step, which I described in the post on how I make Codex review Claude Code's commits https://rayastats.com/en/blog/how-to-make-codex-review-claude-code-commits?utm source=devto&utm medium=article&utm campaign=kit-3wk-2026-10&utm content=weakened-tests . The engine that writes the code often writes the tests too, and a test that already passes on the old code proves nothing about the change. So the writer runs each new test once against the base commit and pastes the failing output into the commit body. That does nothing for an old test that got weaker, which is why the prompt rule matters. A script that reads the diff and flags removed or loosened assertions does not exist. I told Kartik it would be a good addition and that I'd look at it. Today neither the free gate nor the kit has one, so a single weakened assertion gets caught only if the reviewer notices it. review-prompt.md : new tests must fail on the old behaviour, and no test gets weakened or deleted without a stated reason. GATE COUNT RE if you use go test or cargo test. GATE MIN TESTS to your current count if a deleted test should fail the gate. The paid Two Engines Kit https://rayastats.com/en/two-engines-kit?utm source=devto&utm medium=article&utm campaign=kit-3wk-2026-10&utm content=weakened-tests currently v1.0.2 ships a review prompt template whose project rules include "tests must fail on the old behaviour; nothing weakened without reason", and its list of risks to check includes "any weakened test". The Two Engines Kit is built on information gathered since the first Codex and Claude CLIs. Clone two-engines-gate https://github.com/IliyaOblakov/two-engines-gate and run bash demo.sh to watch one full loop with a stand-in reviewer, so no model or account is used. tests/run.sh has the zero-test cases for all six test runners, plus the GATE COUNT RE and GATE MIN TESTS cases.