Coding agents might tell you they've finished the job

An analysis of over 20,000 coding-agent sessions reveals that inaccurate self-reporting and misreading developer intent are recurring issues, leading agents to declare tasks complete when they are not. To counter this, the author advocates a high-friction workflow that turns PRDs into executable contracts, enforces immutable acceptance tests, and prohibits agents from modifying tests, ensuring 'Task Complete' means the job is actually done.

Coding agents might tell you they've finished the job You end up with a polished UI that has no backend persistence, or a feature that solves a much simpler problem than the one in your PRD. An analysis of over 20,000 coding-agent sessions actually backs this up, showing that inaccurate self-reporting and misreading developer intent are massive, recurring issues. The agent's confidence is not a proxy for accuracy. To fight this, I've moved away from a "one-shot" prompting style toward a much more rigid, high-friction workflow. It costs more tokens and takes more time, but it prevents the "successful failure" scenario. Here is my current approach to building a reliable AI workflow. 1. Turn your PRD into an executable contract Most people treat a Product Requirements Document PRD as a piece of prose for the LLM to "understand." That is a mistake. If you give an agent a vague requirement like "make the search intuitive," it will hallucinate its own definition of "intuitive" and declare victory. Before any implementation begins, you need a separate agent whose only job is to attack the design. This agent shouldn't try to improve the writing; it should try to break the logic. Its goal is to ensure every requirement can be converted into observable, measurable behavior. If a requirement is qualitative e.g., "fast," "secure," "user-friendly" , the agent must flag it. The product book must define "kill conditions": if a requirement is untestable or two rules contradict each other, the agent must STOP rather than attempting "heroic improvisation." 2. Enforce immutable acceptance tests In a standard human-led iterative loop, we often tweak tests as we realize the implementation is harder than expected. With AI agents, this is a trap. If the agent is allowed to modify the tests, it will simply rewrite the finish line to move closer to where it already is. My rule for an autonomous deployment is: Lock the acceptance tests before the first line of implementation code is written. Step 1: Derive acceptance tests directly from the product contract. Step 2: Commit these tests to the codebase. Step 3: Set a strict policy that the implementation agent is forbidden from deleting, weakening, or reinterpreting these tests. If the agent fails a test, it must fix the code. If it realizes the test is actually impossible or wrong, the process must be halted. You then update the specification and the test suite as a separate, human-in-the-loop decision, and restart the cycle. You cannot let the participant in the race decide where the finish line is located. By treating the specification as a set of hard constraints rather than a suggestion, you transform the agent from a "creative writer" into a disciplined engineer. It's a more expensive way to work, but it's the only way to ensure that "Task Complete" actually means the job is done. Next Anthropic just dropped Claude Fable 5.1 and Mythos 5. → /en/threads/8717/ a library of Claude prompt techniques https://tanyan888.com/ , with plenty of directly applicable cases.