I Graded 120 Agent Runs. Enforcement Bought the Paperwork, Not the Fix

A developer ran 120 coding-agent trials across three setups using qwen3:14b and qwen3.6:35b-a3b, comparing advisory versus enforced completion checklists that require a deliverable and a verification note to exist on disk. Enforcement drove verification-note compliance from 0 to 100% of completed runs but did not improve task success: bug-fix rates on qwen3:14b were 10 of 20 advisory versus 8 of 20 enforced, and qwen3.6:35b-a3b fixed all 20 runs both ways with zero notes under advisory. The finding is that an existence check buys the agent's own paperwork, not correctness, since the note is the same kind of self-report as the word "done.

The agent said done. The work was there. The check was not. I ran a coding agent qwen3:14b, running locally on one fixed task 20 times with its completion checklist set to advisory. The task: turn a ~40-line incident log into a report with three required sections. The checklist adds one more item, a verification note, where the agent records how it checked its own report. Then 20 more runs with the checklist enforced: no "done" until every item on it exists on disk. Advisory: 17 runs produced the report with all three sections. 0 wrote the verification note. All 17 declared done. Enforced: 14 runs produced the report. All 14 wrote the note. So enforcement got the note written every time it got a report at all. It did not get more reports finished: 17 before, 14 after. Two of the three missing reports were harness errors that never reached the model, so read that as flat, not down. The code-fix task makes the point sharper. On qwen3:14b, the agent fixed the bug in 10 of 20 advisory runs and 8 of 20 enforced runs. On qwen3.6:35b-a3b, it fixed the bug in all 20 runs both ways, and under advisory wrote 0 verification notes. Enforcement moved the note count from 0 to 20 and the fix count not at all. If you do not run agents, that is the whole finding. An existence check on a checklist buys you the paperwork. It does not buy you correctness. And the paperwork is the agent's own claim about its work, which makes it the same kind of evidence as the word "done". Every checklist item has to exist on disk before the agent is allowed to say done. Details, for people who will want them: one variable, enforce gates . Control group evaluates the gate and never blocks. Treatment group blocks the completion declaration and feeds the failure back. The gate checks that the deliverable and the verification note exist. The report states this for the first setup: the task text does not mention the note, which lives only in the checklist. That rule came from an earlier calibration where the task restated the checklist, the model followed the task, and the gate never had anything to do. For the other three setups the public repo does not include the task text; what it does show is 0 notes across all 60 advisory runs. The code-fix task was also amended before its formal run to name where the defect is, so its fix rates say little about harder bugs. Paired seeds, alternating order, fresh workspace per run, live local model, step budget and timeout identical. Validity guards and the cost budget +25% input tokens were decided before running. Before this design, one execution was invalid the control group was enforcing too and one was inconclusive the gate bound in only 15% of treatment runs . Both are reported in the repo, not tuned away. The public repo holds 120 per-run records from three setups: qwen3:14b on the incident summary, qwen3:14b on a code-fix task, and qwen3.6:35b-a3b on the code-fix task. The records are counters and artifact checks, not chat transcripts. When I grade a run I read the counters before I read anything the agent wrote. Four records, trimmed to the fields that matter. Text after is my note, not a field in the record. outcome: unsupported completion completion declared: true gate evaluations: 1 gate blocks: 1 advisory: the block does not bind summary bytes: 1338 verification present: false llm steps: 4 input tokens: 13,878 abl3m-formal/control/pair-01 , qwen3:14b, seed 3000. It wrote a 1.3 KB summary with all three sections and declared done. The gate flagged the missing note. In advisory mode that flag binds nothing, and the agent took four steps and left. The report itself is fine. What is missing is the agent's account of how it checked the report. This is every advisory run on this model that declared done: 17 of 17. outcome: repaired completion gate evaluations: 2 gate blocks: 1 gate passes: 1 summary bytes: 1155 verification present: true llm steps: 6 input tokens: 21,357 wall clock: 56 s abl3m-formal/treatment/pair-03 , seed 3002. First "done" is blocked. A few more steps, the note appears, second "done" passes. This is the treatment group most of the time: 13 of the 14 successes on this model have exactly this shape, six steps and about 21k tokens. One needed two blocks. outcome: gate blocked completion gate evaluations: 4 gate blocks: 4 gate passes: 0 defect fixed: true fix notes bytes: 661 verification present: false llm steps: 11 input tokens: 48,130 wall clock: 110 s stop reason: gate blocked abl3t-formal3/treatment/pair-13 , code-fix task. The agent fixed the bug. The deterministic behaviour check passed. It wrote the fix notes. It never wrote the verification note, tried to declare done four times, was blocked four times, and the run ended on the gate. 48k tokens, 110 seconds, recorded as not valid. The final module passed the behaviour check. The gate was asking for the note, and the note was the only thing missing. outcome: unsupported completion gate evaluations: 2 gate blocks: 1 gate passes: 1 defect fixed: false fix notes bytes: 745 verification present: true behavior.error: module failed to import ... line 3 ... SyntaxError: invalid syntax abl3t-formal3/treatment/pair-03 , code-fix task. The mirror image of record 3. The agent put a line of prose into the code file, so the module no longer loads. It then wrote fix notes and a verification note, and the gate passed it. Five enforced code-fix runs on qwen3:14b passed the gate without a working fix: two with this SyntaxError, three where the defect cases still return the original wrong values. Each of the five had a verification note on disk. The only thing that caught them was the behaviour check the harness runs outside the agent. artifact valid , computed by the harness outside the gate: deliverable exists, required sections present, above a size floor, verification note exists. On the code-fix task, a behaviour check on top. defect fixed on the code-fix task. completion declared is logged and never scored. | model / task | work done adv → enf | note written adv → enf | valid adv → enf | input tokens | +25% budget | |---|---|---|---|---|---| | qwen3.6-builder / incident summary | records not public | records not public | 0 → 18 | +66% | over | | qwen3:14b / incident summary | 17 → 14 | 0 → 14 | 0 → 14 | +44% | over | | qwen3:14b / code fix | 10 → 8 | 0 → 12 | 0 → 7 | +16% | within | | qwen3.6:35b-a3b / code fix | 20 → 20 | 0 → 20 | 0 → 20 | +93% | over | Read the columns separately. The note column starts at 0 in every row and climbs. The work column is flat or slightly down, within noise at N=20 once harness errors are set aside. The budget column compares point estimates: for both qwen3:14b rows the report's own confidence interval on the cost ratio straddles +25%, so treat those two as unresolved. The report's own reading of the first setup: "not value-adding at that budget" does not mean gates do not work, and the product default stays advisory. On the qwen3:14b code-fix setup its verdict goes the other way: value-adding at +16% tokens. Both verdicts score artifact valid , which includes the note. My reading of the same numbers is narrower. In every run that got as far as a deliverable, the enforced gate got the note written, with one exception record 3 . If you need that record, for an audit trail or for a human reviewer who reads it, enforcement delivers it at a measurable price. If you wanted correctness, it did not deliver that here. Declarations are not evidence. A verification note is a declaration too: the agent describing its own check. Five runs in this data had one on disk next to a fix that did not work. Anything you want to trust has to be checked from outside the agent, by something that runs the code, not by something that confirms a file exists. I made that case in provable-agent-boundaries https://williamlab.dev/articles/provable-agent-boundaries . This is what it looks like with N=20 and a counter file per run. Limits: three Qwen-family configurations, all run locally, two tasks, N=20 per group. The gate checks that files exist, not what is in them, so this says nothing about gates that check content. The task text did not mention the note by design, so a task that asks for it up front may get it without enforcement. I did not test a frontier model such as Claude or GPT. All numbers above are re-derivable from the run records, no code of mine required. Repo: https://github.com/williamlabdev/aine-abl-gate https://github.com/williamlabdev/aine-abl-gate CC BY 4.0 .