Three things get said about project rules files for AI coding agents, and they get said everywhere. They are advisory rather than enforcement. Long ones get ignored. Rules buried deep in the file get skipped, so keep it short and move anything that matters into a hook.
I have never seen any of it measured, so I measured it. Twenty checkable rules in a 1,350-line CLAUDE.md, one ticket-sized task, thirty runs across six configurations: the full file preloaded, the file removed entirely, the file on disk with auto- suppressed, nineteen lines plus a Skill, and two more on a stripped-down brief. A deterministic checker reads the resulting file tree.
With the file present, 399 of 400 rule checks passed. Whatever is going wrong with these files, it is not that the model ignores them. The rest of this is about what is going wrong instead.
Two rules out of twenty.
Seventeen of the twenty rules scored five out of five in every configuration, including both arms with no file at all. Only three rules ever failed anywhere across six hundred checks, and two of them account for the entire gap.
The first version of this experiment found no effect at all. Every configuration scored full marks including the ones with no rules file, and I had two articles drafted off it before I found out it was wrong.
Deleting the file from the working tree leaves it in git, and git status announces it by printing D CLAUDE.md in the first command an agent runs to orient itself. Ten of ten runs recovered it, and six said outright that they had read the contract out of HEAD. Two other channels were leaking too: a memory plugin I had installed months earlier and forgotten about, and Claude Code's own per-project memory directory, which I did not know was there. The base repository was rebuilt so the file was never committed at any depth, verified by scanning every blob in the object store.
Then there is the evidence I did not build. The test harness carries a comment pointing at CLAUDE.md. In the arms without the file the model reads that comment, goes looking, and reports that the file does not exist. Four of the ten said so, one noting the harness cites a CLAUDE.md rule pinning something but no such file exists in the repo or its history, so it followed the conventions the committed modules demonstrate instead. That is confirmation from the subject rather than the auditor.
It is also a confound worth naming. Those arms are not a model that never heard of a contract. They are a model told a contract exists and unable to reach it.
Which raises the obvious objection to the experiment. The brief told the model to follow the way the existing utilities are wired in, so of course it copied them. Two of the six configurations ran with that clause removed. The score moved from 94 to 93. One check in a hundred.
Checking turns out to be hard enough that the first careful attempt still came out backwards.
The changelog bullet and the version bump.
Rule 12 says put a bullet under ## Unreleased. The repository says the opposite twice over. That heading has held the placeholder - Nothing yet. for the project's entire history while all eight released sections hold the actual bullets, and the changelog's own header instructs you to cut a new version section and move the patch digit on every merged change.
Every run without the file read that header and followed it. One wrote this about its own work:
CHANGELOG.md + version → 0.4.3, per the changelog's stated rule that the patch digit moves on every merged change.
That is not a model ignoring instructions. It is a model following the only instructions it could see, which were in the repository rather than in the file.
Rule 13 broke the same way, more subtly. The file says bump the patch digit. The repository's history says 0.4.0 was a minor bump because it added new API surface. Three runs added a new public module and picked 0.5.0, one citing that precedent outright. A fourth left the version alone.
Rules 12 and 13 are the only two rules in the set where the file contradicts what the repository demonstrates.
Rules eleven to twenty all sit below line 314, under a heading where the file says of them, in its own words, that in practice they are the ones most often skipped.
Seven of those ten never failed anywhere. Rules 12 and 13 failed only where the file was absent, which cannot be a burial effect, because burial only exists when the file exists.
That leaves one buried-rule failure in a run that had the file: a single missing manifest line. One failure in the fifteen runs that carried the full file. The other fourteen had that rule at exactly the same depth.
Depth explains nothing here. Disagreement with the surrounding code explains all of it.
Rule 19 forbids dependency changes. The task makes one tempting: npm run typecheck fails on a pre-existing error that only @types/node fixes. Of the twenty runs that had rule 19 loaded, seventeen cited it by number and declined.
No run in any arm shipped a dependency change, including the ten that never saw rule 19. Several installed the package with --no-save to run the suites and reverted it, in arms with the rule and arms without. The repository has no dependencies and never has, and every run read that the same way.
The rule did not change the outcome. It supplied language for a decision the repository had already forced.
Go through your file rule by rule and ask one question: would a careful reader of this repo already do this? If yes, the rule is probably inert. Delete it, run a real task twice, and see whether anything changes.
The rules worth keeping are the ones your codebase contradicts, which is uncomfortable, because those are also the ones most likely to be wrong. Rule 12 is arguably the worse convention, and the file never reconciled itself with the eight examples sitting next to it.
One number worth having: the same twenty rules delivered as nineteen lines plus a Skill scored a hundred out of a hundred at $0.696 a run. The full file scored ninety-nine at $1.094, and was dearer on all twenty-five pairwise comparisons with no overlap.
This repository was the best possible case for inference: four worked examples of every convention, matching tests, real history. Eighteen of the twenty rules redundant is a floor, not a ratio. On a greenfield repo the file carries most of the load, and I have not tested that. One task, one repo, one model, five runs an arm, and no statistical testing on the compliance results. The rule 12 split is large enough to be obvious; the rule 13 split is not.
Calling those two a violation also overstates it. The models followed a defensible convention documented in the repository and said which one. A rule set that agreed with the repo would probably have produced the null result honestly.
If you have one of these files, how many lines is it? And has anyone on your team ever checked whether a rule in it contradicts something the codebase already demonstrates? I found mine by accident and I would like a better method than that. I write about architecture and AI engineering for companies that aren’t Google. The newletter lives at mayankkaul.com