Treat an AI coding agent's "done" as a claim

BenchSmoke, a testing outfit for AI coding tools, reports that it verifies every agent's "done" claim with its own scripted checks rather than trusting the agent's final message. In a recent run, an agent built a to-do CLI and declared it finished without executing its own code; BenchSmoke's verification script found two edge bugs — empty input accepted as a task and a wrong error message for non-numeric complete commands — that would have gone unnoticed. The team says a short shell script and a consistent task, with results logged and scored on six criteria, are enough to keep comparisons fair.

When we test AI coding tools at BenchSmoke, we do not treat the agent's final message as proof. We run our own checks. That habit has saved us from shipping a false "it works" more than once. In one recent run, an agent wrote a small to-do CLI, told us the start command, and stopped. It never ran its own code. The app mostly worked when we checked it ourselves: add, complete, delete, and a restart that kept state. Our separate script still found two edge bugs. Empty input was accepted as a task. A non-numeric complete command printed the wrong error. Both would have been invisible if we had stopped at the agent's "done" line. For our standard coding task we always check the same things: add two tasks, mark one complete, delete one, restart the process, and confirm state survived. Same starting prompt for every tool in the category. Same verify steps every time. That is how comparisons stay fair. We write down date, version, plan and price we checked, exact prompts, time to a working result, retry count, what broke, and limits we hit. If it is not in the notes, it does not go in a review. We score six questions from 1 to 5 does it work, time saved, output quality, ease of use, value, reliability , then pick Use it, Watch it, or Skip it. When two verdicts feel close, we take the lower one and explain why. A feature we could not finish on our machine gets marked "not tested," not guessed from the docs. Guessing is how reviews turn into marketing copy. None of this needs a fancy lab. A short shell script and a consistent task beat vibes. If you only click around until the happy path works, you will miss the edges the agent never tried. Our full method is public: How we test https://benchsmoke.com/how-we-test .