Earlier this month Claude Code made a commit in our test app with the message "Limit contact name to 250 chars". The tests passed. Our verification hook ran, passed, and stamped the commit. CI was green.
The name limit did not change. The email limit did.
We were testing a GitHub Action for laravel-ai-verify, a package I maintain that makes coding agents run the checks a change needs before they say "done". The test app is a plain Laravel contact form. The form request had two rules next to each other:
'name' => ['required', 'string', 'max:200'],
'email' => ['required', 'email', 'max:255'],
The intent was to limit the name to 250 characters. The agent did what agents often do with a one-line change: a find-and-replace of max:255 with max:250. There was no max:255 on the name line. There was one on the email line.
Every check we had said yes:
ai:verify wrote a receipt, and the git hook added the trailer: AI-Verify: pass ... checks=4.
The receipt was honest. The checks it lists really ran, on exactly the tree that was committed. What it could not tell anyone is that the commit did something other than what its message said. CodeRabbit caught it in review a day later, as an outside-the-diff comment.
That slip made something obvious that I had been blurring:
Most "verify the agent" setups only answer the first one. Tests catch broken behaviour, not a correct-looking change in the wrong place. And when the subject line says "name", a human reviewer reads the diff expecting the name, which is exactly how this kind of slip gets approved.
Since 0.5.0, the same git hook and the CI check read each commit's subject against its diff. The rule is deliberately narrow:
'name' => [...] untouched, its twin 'email' => [...] edited.200 in the message when the diff writes 250.
It ignores prose, JSON, YAML and HTML files, tests, merges, reverts and commits touching more than 25 files. Agents commit with git commit -m, so the hook can print the warning before the commit exists:
ai-verify: the commit message may not describe this change:
The subject names `name`, but its line app/Http/Requests/StoreContactMessageRequest.php:26 is unchanged; the change went to the similar line 27: `'email' => ['required', 'email', 'max:250'],`
If the change is right, say what it does; if the message is right, change the code (git commit --amend).
The agent reads stderr, so it gets the chance to fix either the code or the message right away. In CI the same finding becomes a warning annotation on the changed line.
Before shipping it we ran it over 937 commits from five of our own PHP repositories. It flagged 23 of them, about 2.5%. One of the 23 was the commit above. That is low enough to leave it on by default as a warning, and it is why the default is warn, not fail: it is a heuristic, and a heuristic that blocks commits gets switched off.
It does not understand intent. A commit that says "fix rounding" and rounds the wrong way passes. A message that is too vague to name anything passes. It catches one specific, very common shape of mistake: the edit landed on the neighbour of the line the message talks about. That shape is common precisely because agents edit by pattern, and similar lines sit next to each other in validation rules, config arrays and route files.
The bigger lesson for me is about what a green check means. "Verified" should say what was verified. Our receipt now says which checks ran on which tree, and the message check says whether the subject matches the diff. Neither says the change is right. That part is still review.
How do you check that an agent's commit does what its message says? Do you rely on a review bot, on humans reading every diff, or on something deterministic that runs before the commit lands?
The package is a Laravel port of the verification step from Semitexa, the PHP framework I build. The message check, the receipts and the Action are in the README.