{"slug": "ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete", "title": "AI Agent Definition of Done: Make Coding Agents Prove the Task Is Complete", "summary": "A developer outlined a model- and framework-agnostic approach for gating AI coding agents with an executable \"definition of done\": a per-task contract specifying allowed paths, acceptance criteria, required evidence, and an independent process that accepts or rejects completion. The contract is paired with a receipt format that records command, exit code, environment, artifact path and timestamp for each criterion, so agents can propose completion but cannot grade their own work. The guide argues that model-generated explanations, unverified screenshots and green test commands without captured output are clues rather than acceptance evidence.", "body_md": "An AI coding agent can change the right-looking files, write a cheerful summary, and still leave the task unfinished. A test may never have run. A migration may work only on an empty database. A visual bug may be hidden outside the agent's viewport. “Done” is a claim; production systems need evidence.\n\nThe fix is not a longer prompt asking the agent to be careful. Give every task an executable definition of done: a small contract that names the allowed change, expected observable outcome, required evidence, and the independent process that accepts or rejects it. The agent may propose completion. It should not grade its own homework.\n\nThis guide shows how to build that gate for coding-agent workflows without tying it to a particular model, IDE, or orchestration framework.\n\nCoding agents optimize for finishing the assigned interaction. They can infer the wrong intent, stop after a partial fix, or interpret a passing command as proof of a broader requirement. Even a careful agent has limited visibility into production data, browser states, deployment configuration, and implicit team conventions.\n\nThis is not an argument against agent autonomy. It is an argument for separating two decisions:\n\nThat separation mirrors normal engineering. A pull request is not merged because the author says it is correct; it is merged after checks and review. The same principle becomes more important when a single agent can edit many files quickly.\n\nRecent agent-evaluation work also measures more than a final answer. NVIDIA describes a hierarchy from benchmarks through trials, tasks, turns, and steps, while trajectory-focused benchmarks inspect tool use, verification attempts, and repository state—not merely a success sentence. [NVIDIA's overview of agent evaluation](https://developer.nvidia.com/blog/how-to-evaluate-ai-agents-from-tool-calls-to-task-completion/) is a useful reminder: completion needs an observable outcome.\n\nBefore the agent receives write access, translate the task into a compact contract. Do not ask it to invent the criteria after it edits code.\n\n```\ntask_id: checkout-tax-042\ngoal: \"Calculate regional tax before rendering the order total\"\nscope:\n  allowed_paths:\n    - apps/checkout/**\n    - packages/tax/**\n  forbidden_paths:\n    - infra/**\n    - db/migrations/**\nacceptance:\n  - id: unit-tax\n    kind: command\n    run: \"pnpm test --filter tax -- --runInBand\"\n    expects: { exit_code: 0 }\n  - id: total-visible\n    kind: browser\n    route: \"/checkout?region=CA\"\n    expects:\n      text: \"Tax\"\n      screenshot: true\n  - id: no-regression\n    kind: command\n    run: \"pnpm lint --filter checkout\"\n    expects: { exit_code: 0 }\npolicy:\n  max_files_changed: 8\n  human_review_required: false\n```\n\nGood criteria have three qualities:\n\nThe contract should also say what is *not* proof. A model-generated explanation, an unverified screenshot, and a green test command with no captured output are useful clues, not acceptance evidence.\n\nStore a receipt for every criterion. This makes the handoff reviewable, lets a failed run resume safely, and provides a concrete artifact when a user reports a regression.\n\n```\n{\n  \"taskId\": \"checkout-tax-042\",\n  \"commit\": \"9ac63c1\",\n  \"criteria\": [\n    {\n      \"id\": \"unit-tax\",\n      \"status\": \"passed\",\n      \"evidence\": {\n        \"kind\": \"command\",\n        \"command\": \"pnpm test --filter tax -- --runInBand\",\n        \"exitCode\": 0,\n        \"stdoutDigest\": \"sha256:...\",\n        \"artifact\": \"artifacts/unit-tax.log\"\n      }\n    },\n    {\n      \"id\": \"total-visible\",\n      \"status\": \"passed\",\n      \"evidence\": {\n        \"kind\": \"browser\",\n        \"url\": \"http://preview.local/checkout?region=CA\",\n        \"screenshot\": \"artifacts/total-visible.png\",\n        \"checkedAt\": \"2026-11-11T10:20:00Z\"\n      }\n    }\n  ]\n}\n```\n\nCapture the command, exit code, environment reference, artifact path, and timestamp. Hash large output instead of placing secrets or full logs in a chat transcript. For browser checks, record the URL, viewport, and screenshot. For API tasks, capture a redacted request shape and response assertion. Evidence should be enough for a reviewer or a later verifier to rerun the check.\n\nDo not confuse “a test exists” with “this change was tested.” The receipt binds a criterion to the tested commit and environment.\n\nThe safest default is a second execution role that has read access plus narrowly scoped test commands, but no write capability, production credentials, or permission to change the contract. It can validate the implementation without fixing it into compliance.\n\nThat role can be a deterministic CI job, a separate agent with constrained tools, or both. Deterministic checks should lead: type checks, unit tests, contract tests, diff policy, and deployment smoke tests. Use an AI verifier where judgment is genuinely needed—for example, assessing whether a screenshot contains a required user flow—but require it to cite its evidence and mark uncertainty.\n\n```\ntype Verdict = \"accepted\" | \"rejected\" | \"needs_review\";\n\nasync function verify(task: Task, receipt: Receipt): Promise<Verdict> {\n  if (!sameCommit(task.proposedCommit, receipt.commit)) return \"rejected\";\n  if (!pathsAllowed(task.scope, await changedPaths(receipt.commit))) return \"rejected\";\n\n  for (const criterion of task.acceptance) {\n    const evidence = receipt.criteria.find((item) => item.id === criterion.id);\n    if (!evidence || evidence.status !== \"passed\") return \"rejected\";\n    if (!(await independentlyRecheck(criterion, evidence))) return \"rejected\";\n  }\n\n  return task.policy.human_review_required ? \"needs_review\" : \"accepted\";\n}\n```\n\nKeep the verifier from rewriting tests, widening the allowed paths, or silently skipping flaky checks. If it detects a failure, its job is to reject with facts: criterion ID, observed result, artifact link, and whether a retry is safe. The implementing agent can then take another attempt under the same contract.\n\nAvoid a Boolean `done`. It hides the difference between an agent that is still working, one that made a plausible proposal, and one whose outcome has been independently established.\n\n| State | Meaning | Who may transition it | \n|---|---|---|\n| `planned` | Contract reviewed; no writes yet | Orchestrator or human | \n| `implementing` | Agent has scoped write access | Implementer | \n| `proposed` | Agent supplied a commit and receipt | Implementer | \n| `verifying` | Evidence is being replayed or checked | CI/verifier | \n| `accepted` | Every required criterion passed | Verifier or human | \n| `rejected` | Evidence contradicts a criterion | Verifier | \n| `needs_review` | Evidence is incomplete or judgment-sensitive | Verifier | \n\nOnly `accepted` should unlock a merge, deployment step, billing event, or a message that tells a customer the task is complete. `needs_review` is a healthy outcome, not an error; it stops the system from pretending ambiguity is success.\n\nThe fastest way to discover a weak ticket is to try to write the verifier first. If nobody can state what command, request, or visible condition proves success, the agent will have to guess too. That guess often becomes a broad patch and a slow review cycle.\n\nFor each criterion, ask four questions:\n\nThis exercise should not turn a normal bug fix into a formal specification project. It is a short translation from natural language into proof. The contract can evolve when the team learns something real, but a changed contract is a new review event—not an invisible way to convert a failed run into a pass.\n\nTests are essential, but a good AI agent definition of done includes the other forms of evidence a task needs.\n\nRequire a browser check at the relevant viewport, an accessibility assertion, and a screenshot stored outside the model context. If the change affects a critical funnel, use a short human acceptance step. A visually plausible image is not proof that keyboard navigation or loading states work.\n\nUse a disposable dataset or a transaction that can roll back. Verify row counts, constraints, idempotency, and a downgrade path. Never let an agent call a production migration “done” because a locally generated SQL file looks reasonable.\n\nRecord the sandbox account, fixture IDs, rate-limit result, and webhook outcome. For a write-capable integration, verify both the intended side effect and the absence of a duplicate side effect. This pairs naturally with idempotency and reconciliation controls.\n\nAdd a policy check before functional testing: forbidden dependencies, secret scanning, permission diff, and egress rules. A feature can work and still fail its definition of done if it expands the blast radius.\n\nAn agent can accidentally optimize for the check rather than the user outcome. Treat the acceptance system like production code.\n\nThis does not make tests impossible to fool. It makes cheating, accidental test weakening, and unsupported “done” claims visible and costly enough to catch before release.\n\nDo not turn every tiny change into a ceremony. Start with an agent workflow that already causes review churn: a bug-fix queue, UI fixes, or integration changes. Build three to five criteria per task and make one check independent. Then track:\n\nHigh rejection rates can mean agents need better context, but they can also reveal vague tickets, brittle tests, or missing fixtures. Treat the gate as a diagnostic tool. Tighten criteria only when they map to a meaningful user or engineering outcome.\n\nFor related controls, combine this pattern with a code-review evidence packet, [invariant tests](https://dev.to/jackm-singularity/ai-code-invariant-tests-catch-regressions-that-green-test-suites-miss-5fbk), and trace replay for failures that escaped. The acceptance gate answers a narrower, crucial question: may this particular agent task truthfully be called complete?\n\nIt is a machine-checkable contract that defines the requested outcome, scope limits, required evidence, and acceptance authority for an agent task. It separates an agent's completion proposal from verified acceptance.\n\nNot always. Tests may be incomplete, skipped, or unrelated to the user-visible result. Pair them with scope checks, fresh command output, and task-specific evidence such as an API assertion, browser check, or migration verification.\n\nIt can run local checks while implementing, but the final acceptance decision should be independently reproducible. A CI job or read/test-only verifier reduces self-confirmation and protects the contract from being rewritten to fit the patch.\n\nUse the fewest criteria that establish the outcome and protect the risky boundaries—often three to five. More checks are not better if they do not correspond to an observable requirement.\n\nMove the task to `needs_review`, not `accepted`. Preserve the partial receipt, state what is missing, and route it to a human or a safe follow-up attempt. Ambiguity should be visible.\n\nYes. A single coding agent plus CI can use the same contract and receipt. The important separation is between implementation and acceptance authority, not the number of models involved.", "url": "https://wpnews.pro/news/ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete", "canonical_source": "https://dev.to/jackm-singularity/ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete-2ejm", "published_at": "2026-11-11 08:44:24+00:00", "updated_at": "2026-11-11 08:52:13.097708+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["NVIDIA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete", "markdown": "https://wpnews.pro/news/ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete.md", "text": "https://wpnews.pro/news/ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete.txt", "jsonld": "https://wpnews.pro/news/ai-agent-definition-of-done-make-coding-agents-prove-the-task-is-complete.jsonld"}}