A Coding Agent's Green Check Is a Coverage Claim

A developer outlined a review protocol for coding agents that treats a green test check as a coverage claim rather than proof of correctness, since results can come from a stale commit or a suite that no longer tests the requested behavior. The proposed approach records the reviewed commit or immutable diff, the command and its output, any skipped or removed checks, and behavior inspected outside the tests, with unexercised paths marked incomplete. The developer cautions that letting an agent write its own acceptance test after implementing a change can duplicate the same misunderstanding in both code and test.

An agent says the tests passed. The screenshot is green. But the result could be from yesterday's commit, and the test for the behavior you requested might not even be in the suite anymore. A green check tells you about the checks that ran. It says nothing about the behavior nobody checked. Suppose you ask a coding agent to add a settings toggle. Before it edits the app, write down what you'd expect to see: the default value on first load, the saved value after reopening settings, and the state after reset. These are proposed acceptance checks for an example change, not a test of any particular product. After the agent finishes, inspect the running behavior or stored state, not just the final message. A description of a file move is not evidence that the file ended up at its destination. The same gap applies to a toggle that looks right in a screenshot but does not persist. Independent checks needn't be elaborate; they need to touch the thing the task was meant to change. Letting the agent write the acceptance test after implementing the toggle is risky. If it misunderstood the default, it can put the same mistake in the code and the test. Keep at least one expected outcome outside that loop, even if it's a human-reviewed assertion written before the edit. A test result without a revision could belong to different code. Record the commit or immutable diff, the command and its output. Then inspect whether relevant checks were skipped, removed, or weakened. A focused test may pass while the broader suite remains unknown. Say both things. Here is a proposed review card for that hypothetical settings change. Fill it with actual evidence from your own repository rather than copying these placeholders into an approval: Intent: default off; saved choice survives reopening; reset returns to default Revision reviewed: <commit or immutable diff identifier Files changed: <reviewed file list Checks expected before editing: <default / persist / reset checks Command and result on that revision: <command + captured output, or NOT RUN Checks changed or skipped: <list, or explicitly none after inspection Behavior inspected outside tests: <observation, or NOT CHECKED Verdict: accepted / needs work / incomplete If the reset path was never exercised, the verdict on that requirement is incomplete. It does not become accepted because the other tests passed. If a test disappeared in the same diff, investigate the removal before taking the surviving green tests at face value. This is a review protocol, not a claim that any particular agent or tool automatically enforces it. The toggle might work while the agent logs a token, touches an unrelated file, or requests access it did not need. Those are different acceptance questions. Check the diff for scope; for sensitive flows, use non-real planted credentials and watch the paths the agent actually takes. Keep a record of the paths the check never reached. A clean result on an exercised path is worth recording. It is not a certificate that every possible path is safe. Likewise, a tool that compares a completion claim to session evidence cannot infer an unobserved user requirement. The evidence has to match the question you are asking. Skip the card for a one-word documentation edit. For stored preferences, permissions, or production data, write the assertion down and keep the test receipt with the review. If a path wasn't checked, mark it incomplete. The next reviewer can run that check; they can't recover it from a confident closing paragraph.