How to Review an AI-Suggested Test Patch Without Trusting the Diff Blindly

Angela (Lo An) published a review checklist for evaluating AI-suggested test patches, emphasizing that test diffs change what reviewers can believe about behavior. The guide provides concrete criteria—such as checking whether production code fixes the root cause and whether assertions are weakened—and recommends labeling PRs and reviewing test changes first to avoid accepting false greens. It also advises rejecting silent skips and mass snapshot updates, and suggests a template for requesting changes.

When a coding agent “fixes CI” by editing tests, the diff can look helpful and still destroy signal. This guide is a review checklist for testers and reviewers who need to decide: merge, request changes, or reject . No tool vendor required. No marketing CTA. Production diffs change behavior. Test diffs change what you are allowed to believe about behavior. An agent optimizing for green can: toBeTruthy vs exact value All five can produce a green pipeline. Only some preserve product truth. In the PR, force a label: agent-prod-fix — production/config only agent-test-edit — any test file touched agent-ci-config — pipeline definition changed If both prod and tests changed, review tests first . Otherwise you may accept a green that only exists because the alarm was muted. For every removed or changed assertion: | Question | Pass looks like | Fail looks like | |---|---|---| | What product rule did this assert? | Reviewer can state it in one sentence | Nobody knows why it existed | | Is the bug fixed in production code? | Prod diff addresses root cause | Only test changed | | Did we keep an equivalent guard? | New assert is same or stricter | Assert deleted or weakened | | Is the new matcher precise? | Exact value / structured match | truthy / defined / empty catch | Rule of thumb: if the failing assert disappears and production code barely changes, treat as false green until proven otherwise. Skips are sometimes correct. Unowned skips are how coverage dies. Require: Reject silent xit / skip / @Disabled added by an agent “to unblock main.” Snapshot updates are high-risk agent edits. Ask: Mass snapshot refresh on an agent PR is a smell. Prefer small, explained updates. If the agent edits workflow YAML / pipeline config: continue-on-error ? Config edits that make red jobs invisible are merge blockers. Use this when time is short: test / snapshots / .github/workflows ? skip , xit , toBeTruthy , any , pass , deleted expect . Diff A often OK : production retry logic fixed; test now expects eventual success with the same business rule; assert still checks the user-visible outcome. Diff B reject : production unchanged; test deletes the assert that caught a null; pipeline green. Same “CI fixed” narrative. Opposite review outcome. Be specific. Help the next human or the next agent prompt . Request changes: this PR removes the assertion that validated X. Please either 1 fix production so X holds, or 2 replace with an equivalent assert, or 3 skip with ticket ABC and owner @name. Green alone is not enough. Trust the signal , not the author—human or model. If your review process only reads production diffs and glances at “tests updated,” agent workflows will teach your suite to agree with bugs. Make test-diff review a first-class gate. Author: Angela Lo An . Written for testers and reviewers adopting AI-assisted CI repair. Pair with an evidence-pack PR template for best results.