Evidence Gates for AI Coding Agents in CI — Recoverable Merge over Mean Time to Green

Angela (Lo An), a practitioner in CI and testing, argues that coding agents' ability to quickly fix failing CI logs and open pull requests can be dangerous if teams treat a green pipeline as proof of safety. She emphasizes that recoverable merge and evidence packs matter more than mean time to green, and provides a checklist for platform engineers, SDEs, and testers to ensure agent speed without silent quality loss. The article outlines risk tiers, evidence requirements, and the principle that 'agent proposes, humans dispose' to maintain accountability.

Recoverable merge matters more than mean time to green. Coding agents can read failing CI logs and open pull requests in minutes. That is useful for mechanical failures. It becomes dangerous when teams treat a green pipeline as proof that an agent-authored change is safe to land. This article is a practical checklist for platform engineers, SDEs, and testers who want agent speed without silent quality loss. It is not a product pitch. Most “AI fixes CI” demos optimize one metric: time until the pipeline is green. That metric ignores three questions: If those questions are unanswered, the agent is not repairing engineering—it is optimizing a dashboard. Minimum bar before an agent may open a mergeable PR: Rollback - Previous green SHA on target: - Revert plan one line : Trigger - Failing CI run URL/id: If the team cannot name the previous green SHA, they are not ready for agent auto-merge. “I ran the tests” is not evidence. Merge criteria should require an evidence pack : | Field | Example | |---|---| | Commands | npm run test:unit && npm run test:api | | Results | unit 412 pass; api 38 pass | | Uncovered scope | e2e nightlies not run | | Tests edited? | YES — login.spec.ts tightened assertion | Treat test diffs as higher risk than production diffs of similar size. Review for: A green suite that no longer guards the bug is a false green . | Risk tier | Examples | Agent may | Merge requires | |---|---|---|---| | Low | format, comment typos | open PR | optional policy auto-merge | | Medium | test logic, business code | open PR | human owner | | High | auth, payments, migrations, release flags | draft only | human + often second reviewer | Raise automation by classifying Low risk accurately—not by expanding silent merges. Score these separately from happy-path repair rate: Ask: did gates block merge , even if the model attempted the unsafe action? Evidence Pack agent-assisted change - Failing run id linked - Branch-only change no direct default push - File list attached - Tests edited? yes/no + paths - Exact commands re-run - Uncovered scope stated - Rollback one-liner + previous green SHA - Risk tier: Low / Medium / High Paste it into your PR template. Block merge when empty—the same way you block PRs without reproduction steps. Durable pattern: agent proposes, humans dispose —especially on Medium/High tiers. Speed is real. Accountability is still human. Until rollback, reviewable evidence, and merge authority are explicit, automatic landing should not be the default—no matter how impressive the mean time to green looks in a demo. Author: Angela Lo An . Practitioner notes for CI + testing teams. Comments welcome on what your team already requires in PR templates.