The commit passed every check. It still changed the wrong line.

The maintainer of laravel-ai-verify, a Laravel package that makes coding agents run pre-commit checks, added a commit-message-versus-diff check after a Claude Code agent changed an email max-length rule instead of the intended name rule while every test, hook and CI check passed. Since version 0.5.0, the git hook and CI check compare each commit subject against its diff and emit a warning when the edit lands on a neighbouring line, flagging 23 of 937 commits (about 2.5%) across five PHP repositories, including the original slip.

Earlier this month Claude Code made a commit in our test app with the message "Limit contact name to 250 chars". The tests passed. Our verification hook ran, passed, and stamped the commit. CI was green. The name limit did not change. The email limit did. We were testing a GitHub Action for laravel-ai-verify https://github.com/semitexa/laravel-ai-verify , a package I maintain that makes coding agents run the checks a change needs before they say "done". The test app is a plain Laravel contact form. The form request had two rules next to each other: js 'name' = 'required', 'string', 'max:200' , 'email' = 'required', 'email', 'max:255' , The intent was to limit the name to 250 characters. The agent did what agents often do with a one-line change: a find-and-replace of max:255 with max:250 . There was no max:255 on the name line. There was one on the email line. Every check we had said yes: ai:verify wrote a receipt, and the git hook added the trailer: AI-Verify: pass ... checks=4 . The receipt was honest. The checks it lists really ran, on exactly the tree that was committed. What it could not tell anyone is that the commit did something other than what its message said. CodeRabbit caught it in review a day later, as an outside-the-diff comment. That slip made something obvious that I had been blurring: Most "verify the agent" setups only answer the first one. Tests catch broken behaviour, not a correct-looking change in the wrong place. And when the subject line says "name", a human reviewer reads the diff expecting the name, which is exactly how this kind of slip gets approved. Since 0.5.0, the same git hook and the CI check read each commit's subject against its diff. The rule is deliberately narrow: 'name' = ... untouched, its twin 'email' = ... edited. 200 in the message when the diff writes 250 . It ignores prose, JSON, YAML and HTML files, tests, merges, reverts and commits touching more than 25 files. Agents commit with git commit -m , so the hook can print the warning before the commit exists: ai-verify: the commit message may not describe this change: The subject names name , but its line app/Http/Requests/StoreContactMessageRequest.php:26 is unchanged; the change went to the similar line 27: 'email' = 'required', 'email', 'max:250' , If the change is right, say what it does; if the message is right, change the code git commit --amend . The agent reads stderr, so it gets the chance to fix either the code or the message right away. In CI the same finding becomes a warning annotation on the changed line. Before shipping it we ran it over 937 commits from five of our own PHP repositories. It flagged 23 of them, about 2.5%. One of the 23 was the commit above. That is low enough to leave it on by default as a warning, and it is why the default is warn , not fail : it is a heuristic, and a heuristic that blocks commits gets switched off. It does not understand intent. A commit that says "fix rounding" and rounds the wrong way passes. A message that is too vague to name anything passes. It catches one specific, very common shape of mistake: the edit landed on the neighbour of the line the message talks about. That shape is common precisely because agents edit by pattern, and similar lines sit next to each other in validation rules, config arrays and route files. The bigger lesson for me is about what a green check means. "Verified" should say what was verified. Our receipt now says which checks ran on which tree, and the message check says whether the subject matches the diff. Neither says the change is right. That part is still review. How do you check that an agent's commit does what its message says? Do you rely on a review bot, on humans reading every diff, or on something deterministic that runs before the commit lands? The package is a Laravel port of the verification step from Semitexa https://semitexa.com/?utm source=devto&utm medium=post&utm campaign=receipt-vs-claim , the PHP framework I build. The message check, the receipts and the Action are in the README https://github.com/semitexa/laravel-ai-verify .