I Graded 120 Agent Runs. Enforcement Bought the Paperwork, Not the Fix

The agent said done. The work was there. The check was not.

I ran a coding agent (qwen3:14b, running locally) on one fixed task 20 times with its completion checklist set to advisory. The task: turn a ~40-line incident log into a report with three required sections. The checklist adds one more item, a verification note, where the agent records how it checked its own report. Then 20 more runs with the checklist enforced: no "done" until every item on it exists on disk.

Advisory: 17 runs produced the report with all three sections. 0 wrote the verification note. All 17 declared done.

Enforced: 14 runs produced the report. All 14 wrote the note.

So enforcement got the note written every time it got a report at all. It did not get more reports finished: 17 before, 14 after. Two of the three missing reports were harness errors that never reached the model, so read that as flat, not down.

The code-fix task makes the point sharper. On qwen3:14b, the agent fixed the bug in 10 of 20 advisory runs and 8 of 20 enforced runs. On qwen3.6:35b-a3b, it fixed the bug in all 20 runs both ways, and under advisory wrote 0 verification notes. Enforcement moved the note count from 0 to 20 and the fix count not at all.

If you do not run agents, that is the whole finding. An existence check on a checklist buys you the paperwork. It does not buy you correctness. And the paperwork is the agent's own claim about its work, which makes it the same kind of evidence as the word "done".

Every checklist item has to exist on disk before the agent is allowed to say done.

Details, for people who will want them: one variable, enforce_gates. Control group evaluates the gate and never blocks. Treatment group blocks the completion declaration and feeds the failure back. The gate checks that the deliverable and the verification note exist. The report states this for the first setup: the task text does not mention the note, which lives only in the checklist. That rule came from an earlier calibration where the task restated the checklist, the model followed the task, and the gate never had anything to do. For the other three setups the public repo does not include the task text; what it does show is 0 notes across all 60 advisory runs. The code-fix task was also amended before its formal run to name where the defect is, so its fix rates say little about harder bugs. Paired seeds, alternating order, fresh workspace per run, live local model, step budget and timeout identical. Validity guards and the cost budget (+25% input tokens) were decided before running. Before this design, one execution was invalid (the control group was enforcing too) and one was inconclusive (the gate bound in only 15% of treatment runs). Both are reported in the repo, not tuned away.

The public repo holds 120 per-run records from three setups: qwen3:14b on the incident summary, qwen3:14b on a code-fix task, and qwen3.6:35b-a3b on the code-fix task. The records are counters and artifact checks, not chat transcripts. When I grade a run I read the counters before I read anything the agent wrote. Four records, trimmed to the fields that matter. Text after # is my note, not a field in the record.

outcome:             unsupported_completion
completion_declared: true
gate_evaluations: 1  gate_blocks: 1   # advisory: the block does not bind
summary_bytes: 1338  verification_present: false
llm_steps: 4  input_tokens: 13,878

abl3m-formal/control/pair-01, qwen3:14b, seed 3000.

It wrote a 1.3 KB summary with all three sections and declared done. The gate flagged the missing note. In advisory mode that flag binds nothing, and the agent took four steps and left. The report itself is fine. What is missing is the agent's account of how it checked the report. This is every advisory run on this model that declared done: 17 of 17.

outcome:          repaired_completion
gate_evaluations: 2  gate_blocks: 1  gate_passes: 1
summary_bytes: 1155  verification_present: true
llm_steps: 6  input_tokens: 21,357  wall_clock: 56 s

abl3m-formal/treatment/pair-03, seed 3002.

First "done" is blocked. A few more steps, the note appears, second "done" passes. This is the treatment group most of the time: 13 of the 14 successes on this model have exactly this shape, six steps and about 21k tokens. One needed two blocks.

outcome:          gate_blocked_completion
gate_evaluations: 4  gate_blocks: 4  gate_passes: 0
defect_fixed: true  fix_notes_bytes: 661  verification_present: false
llm_steps: 11  input_tokens: 48,130  wall_clock: 110 s
stop_reason:      gate_blocked

abl3t-formal3/treatment/pair-13, code-fix task.

The agent fixed the bug. The deterministic behaviour check passed. It wrote the fix notes. It never wrote the verification note, tried to declare done four times, was blocked four times, and the run ended on the gate. 48k tokens, 110 seconds, recorded as not valid. The final module passed the behaviour check. The gate was asking for the note, and the note was the only thing missing.

outcome:          unsupported_completion
gate_evaluations: 2  gate_blocks: 1  gate_passes: 1
defect_fixed: false  fix_notes_bytes: 745  verification_present: true
behavior.error: module failed to import ... line 3 ... SyntaxError: invalid syntax

abl3t-formal3/treatment/pair-03, code-fix task.

The mirror image of record 3. The agent put a line of prose into the code file, so the module no longer loads. It then wrote fix notes and a verification note, and the gate passed it. Five enforced code-fix runs on qwen3:14b passed the gate without a working fix: two with this SyntaxError, three where the defect cases still return the original wrong values. Each of the five had a verification note on disk. The only thing that caught them was the behaviour check the harness runs outside the agent.

artifact_valid, computed by the harness outside the gate: deliverable exists, required sections present, above a size floor, verification note exists. On the code-fix task, a behaviour check on top.defect_fixed on the code-fix task.completion_declared is logged and never scored.

Read the columns separately. The note column starts at 0 in every row and climbs. The work column is flat or slightly down, within noise at N=20 once harness errors are set aside. The budget column compares point estimates: for both qwen3:14b rows the report's own confidence interval on the cost ratio straddles +25%, so treat those two as unresolved.

The report's own reading of the first setup: "not value-adding at that budget" does not mean gates do not work, and the product default stays advisory. On the qwen3:14b code-fix setup its verdict goes the other way: value-adding at +16% tokens. Both verdicts score artifact_valid, which includes the note. My reading of the same numbers is narrower. In every run that got as far as a deliverable, the enforced gate got the note written, with one exception (record 3). If you need that record, for an audit trail or for a human reviewer who reads it, enforcement delivers it at a measurable price. If you wanted correctness, it did not deliver that here.

Declarations are not evidence. A verification note is a declaration too: the agent describing its own check. Five runs in this data had one on disk next to a fix that did not work. Anything you want to trust has to be checked from outside the agent, by something that runs the code, not by something that confirms a file exists. I made that case in provable-agent-boundaries. This is what it looks like with N=20 and a counter file per run.

Limits: three Qwen-family configurations, all run locally, two tasks, N=20 per group. The gate checks that files exist, not what is in them, so this says nothing about gates that check content. The task text did not mention the note by design, so a task that asks for it up front may get it without enforcement. I did not test a frontier model such as Claude or GPT.

All numbers above are re-derivable from the run records, no code of mine required.

Repo: https://github.com/williamlabdev/aine-abl-gate (CC BY 4.0).