When a coding agent “fixes CI” by editing tests, the diff can look helpful and still destroy signal. This guide is a review checklist for testers and reviewers who need to decide: merge, request changes, or reject.
No tool vendor required. No marketing CTA.
Production diffs change behavior. Test diffs change what you are allowed to believe about behavior.
An agent optimizing for green can:
toBeTruthy()
vs exact value) All five can produce a green pipeline. Only some preserve product truth.
In the PR, force a label:
agent-prod-fix
— production/config only
agent-test-edit
— any test file touched
agent-ci-config
— pipeline definition changed If both prod and tests changed, review tests first. Otherwise you may accept a green that only exists because the alarm was muted.
For every removed or changed assertion:
| Question | Pass looks like | Fail looks like |
|---|---|---|
| What product rule did this assert? | Reviewer can state it in one sentence | Nobody knows why it existed |
| Is the bug fixed in production code? | Prod diff addresses root cause | Only test changed |
| Did we keep an equivalent guard? | New assert is same or stricter | Assert deleted or weakened |
| Is the new matcher precise? | Exact value / structured match | |
truthy / defined / empty catch |
Rule of thumb: if the failing assert disappears and production code barely changes, treat as false green until proven otherwise.
Skips are sometimes correct. Unowned skips are how coverage dies.
Require:
Reject silent xit
/ skip
/ @Disabled
added by an agent “to unblock main.”
Snapshot updates are high-risk agent edits.
Ask:
Mass snapshot refresh on an agent PR is a smell. Prefer small, explained updates.
If the agent edits workflow YAML / pipeline config:
continue-on-error
? Config edits that make red jobs invisible are merge blockers.
Use this when time is short:
*test*
/ __snapshots__
/ .github/workflows
?
skip
, xit
, toBeTruthy
, any(
, pass(
, deleted expect
. Diff A (often OK): production retry logic fixed; test now expects eventual success with the same business rule; assert still checks the user-visible outcome.
Diff B (reject): production unchanged; test deletes the assert that caught a null; pipeline green.
Same “CI fixed” narrative. Opposite review outcome.
Be specific. Help the next human (or the next agent prompt).
Request changes: this PR removes the assertion that validated X.
Please either (1) fix production so X holds, or (2) replace with an
equivalent assert, or (3) skip with ticket ABC and owner @name.
Green alone is not enough.
Trust the signal, not the author—human or model.
If your review process only reads production diffs and glances at “tests updated,” agent workflows will teach your suite to agree with bugs. Make test-diff review a first-class gate.
Author: Angela (Lo An). Written for testers and reviewers adopting AI-assisted CI repair. Pair with an evidence-pack PR template for best results.