How to Catch False “Done” Claims in AI-Generated Code

A developer created a method to catch false 'done' claims in AI-generated code by requiring specific evidence such as changed-file lists and test commands. The approach helps reviewers distinguish plausible statements from supported ones, using a review loop that flags missing or conflicting evidence. The developer also offers a free tool called the AI Completion Evidence Auditor Lite to automate this process.

AI-generated code can sound finished long before it is ready to accept. A message such as “tests passed” is a claim, not a verdict. Before accepting delivery, ask a more useful question: what evidence actually supports the claim that the requested work is complete? A passing test can be real and still be incomplete. It may cover only a narrow helper, skip the reported failure, use an old output, or test behavior that is adjacent to rather than identical to the requested change. A green result does not tell you which files changed, whether the relevant command ran, or whether the output belongs to the current work. The goal is not to distrust every report. It is to separate a plausible statement from a supported one. A simple review loop makes that separation practical: This keeps a review focused. Instead of arguing about confidence, you can identify the next missing piece of evidence. A changed-file list tells you whether the reported work touched the expected area. It can also reveal scope drift: a tiny fix that unexpectedly changes deployment settings, unrelated dependencies, or authentication code deserves a closer look. Useful test evidence includes the command that ran, the result, and enough context to connect it to the current change. “All tests passed” is less useful than a specific command and its output. The strongest evidence also makes clear which scenario was exercised. Missing evidence is often the whole story. If a report says a browser flow is fixed but supplies only a unit test, the relevant end-to-end evidence is still missing. If a report claims no regressions but only one focused test ran, broader coverage may remain unproven. Look for conflicts instead of averaging them away. A success message beside a failing log, a claim that nothing changed beside a diff, or a test output from before the latest edit should move the verdict away from PROVEN until the contradiction is resolved. Imagine an assistant reports: “The password-reset redirect is fixed and tests passed.” The supplied record contains one unit test for URL construction, but no changed-file list, no browser or route test, and no output for the reset flow. The correct verdict is UNPROVEN . That does not say the fix is wrong. It says the available evidence does not yet justify accepting the full completion claim. The next action is clear: provide the changed files and run a test that exercises the reset flow itself. An evidence check is not a substitute for engineering judgment. It helps a reviewer see what is supported, what is missing, and what conflicts with the report. That makes it easier to ask for a narrow follow-up instead of reopening the entire task. There is one important limitation: a tool that reviews an evidence packet does not re-run commands or independently verify test output supplied by a user. A log can be organized and challenged, but it does not become trustworthy merely because it was pasted into a report. When the risk is high, the right next step may still be to reproduce the result in a controlled environment. If you want a structured way to label completion claims before you accept delivery, try the free Auditor. It is designed to surface unsupported “done” statements, missing proof, and contradictions without pretending that an unverified record is a completed result. Get the free AI Completion Evidence Auditor Lite: https://frankster8205.gumroad.com/l/ai-completion-evidence-auditor-lite?src=devto us article1 v1 https://frankster8205.gumroad.com/l/ai-completion-evidence-auditor-lite?src=devto us article1 v1