Our CLAUDE.md, three rules files, nine skills and one CLI reference contain 707 pnpm <name> references to 190 unique names, checked against 199 scripts in package.json: today, 0 are missing. Git history tells a less tidy story. One skill file pointed at a deleted command for 24 days while 28 commit-time shape tests passed on 142 commits that touched rules files, 10 of them on that very file.
Two readers asked variations of the same question after our last piece on commit-time tests for rules files. One: "Are the file-shape checks enough to catch a rule that reads correctly but no longer matches what the code does?" The other: "What happens when the test breaks and the rule stays stale anyway?" We did not have a measured answer, so this article is the measurement. Everything below was run on our own repository on 2026-09-22 with Claude Code 2.1.278; the numbers are from that run.
Our rules files are dense with commands. CLAUDE.md tells the agent which command to run at the start of a turn, which one records a decision, which one pushes. The skills go further: a skill body is mostly "run this, read that field, then run this". A CLI reference file lists every command with its input and output shape. That is exactly the kind of instruction the official memory page recommends ("Run npm test before committing" instead of "Test your changes"), and it has a cost that the same page names in the paragraph on consistency:
"Periodically" is doing a lot of work in that sentence. A rule that says pnpm append-weekly-analysis is a perfectly well-formed instruction. It has the right frontmatter, it is under the line limit, it has a description. It just names a script that was deleted three weeks ago. The agent will try it, get a package manager error, and then either improvise or stop. Neither outcome is visible in a shape test.
Here is the whole thing. Drop it in the repo root as check-script-refs.sh. It reads package.json, walks the Markdown files you point it at (default: CLAUDE.md, .claude/, docs/), pulls every pnpm x, npm run x and yarn x, and prints the names that are not scripts, with file and line. It exits 1 when there is at least one, so it can sit in a pre-commit hook or CI job unchanged.
#!/usr/bin/env bash
set -euo pipefail
targets=("$@"); [ $# -eq 0 ] && targets=(CLAUDE.md .claude docs)
find "${targets[@]}" \( -name node_modules -o -name worktrees \) -prune -o -name '*.md' -print 2>/dev/null | sort | node -e "$(cat <<'JS'
const fs = require('fs');
const scripts = new Set(Object.keys(JSON.parse(fs.readFileSync('package.json', 'utf8')).scripts || {}));
const builtin = new Set(['install', 'add', 'run', 'exec', 'dlx', 'test', 'build', 'start', 'lint', 'i', 'ci']);
const re = /\b(?:pnpm|npm run|yarn)\s+([A-Za-z][\w:.-]*)/g;
let refs = 0, files = 0; const seen = new Map(), missing = new Map();
for (const file of fs.readFileSync(0, 'utf8').split('\n').filter(Boolean)) {
files++;
fs.readFileSync(file, 'utf8').split('\n').forEach((line, i) => {
for (const m of line.matchAll(re)) {
const name = m[1]; refs++; seen.set(name, (seen.get(name) || 0) + 1);
if (!scripts.has(name) && !builtin.has(name)) missing.set(name, [...(missing.get(name) || []), `${file}:${i + 1}`]);
}
});
}
console.log(`${files} markdown files, ${refs} script references, ${seen.size} unique names, ${scripts.size} scripts in package.json`);
for (const [name, where] of missing) console.log(`MISSING ${name} <- ${where.slice(0, 3).join(', ')}${where.length > 3 ? ` (+${where.length - 3})` : ''}`);
console.log(missing.size ? `${missing.size} name(s) not in package.json` : 'ok: every referenced name exists');
process.exit(missing.size ? 1 : 0);
JS
)"
Twenty-six lines. The builtin set is the list of package-manager subcommands that are not scripts (pnpm install, pnpm test and so on); extend it if your docs mention others. The -prune on worktrees is there because Claude Code's agent worktrees live under .claude/worktrees/ and contain a full copy of the repository; our first run without it scanned 12,800 Markdown files and reported 25 missing names, 19 of which came only from those snapshots and the product fixture files inside them. Scope the input to files that actually load into the agent's context and the noise disappears.
Run on the files that load into context in our repository today, the output is:
$ bash check-script-refs.sh CLAUDE.md .claude/rules .claude/skills docs/cli-reference.md
14 markdown files, 707 script references, 190 unique names, 199 scripts in package.json
ok: every referenced name exists
Fourteen files: one CLAUDE.md, three files in .claude/rules/, nine SKILL.md files, one CLI reference. The reference file alone accounts for 202 of the 707 references; the biggest skill has 103. Zero missing today. If we had stopped here, the answer to the first reader would have been "the shape tests seem fine", which is the wrong answer.
Run with the default targets, which include the whole docs/ folder, it reports 6 missing names: an example pnpm foo in a changelog, and five names that were removed or never built but are recorded as such in the changelog, an architecture-decisions file and an archived copy of an old CLAUDE.md. Those are history, not instructions, and they are a reminder that the check's hits need a human to read them: the tool cannot tell "run this" from "we deleted this".
To find out whether "zero today" was luck, we replayed the check across history. The script for that is longer than 30 lines, so here is the method rather than the code: list every commit that touched CLAUDE.md, .claude/, the CLI reference or package.json (399 commits); at each one, read package.json from that commit with git show <commit>:package.json, grep the rules files at that commit with git grep -o -E "pnpm [a-z][a-z0-9-]*" <commit> -- CLAUDE.md .claude docs/cli-reference.md, and record every name that is missing, together with the first commit where it stops being missing. That gives episodes with a start date, an end date and a gap in days. We found 9 episodes; 5 lasted 0.0 days (a documentation commit landed seconds before the package.json commit that added the script, an ordering artefact). The other 4 are three real stories.
Episode 1: 24 days, the real case. On 2026-08-25 a commit removed a script that appended a cross-theme analysis section to the weekly report, after the owner approved dropping that section. The same commit edited CLAUDE.md, the weekly skill's SKILL.md, the CLI reference and the skill's reference.md, and added a note at the top of the relevant section saying the section was discontinued. It removed the instruction from SKILL.md and from the reference table. It left two lines in reference.md untouched: line 165 still said to write two or more themes and record them with pnpm append-weekly-analysis /tmp/analysis.json, and line 176 still explained what happens when that command throws. The file was a skill's supporting file, the kind the skills documentation describes as only on demand ("Unlike CLAUDE.md content, a skill's body loads only when it's used, so long reference material costs almost nothing until you need it"). It was needed every Monday.
The reference stayed until 2026-09-18, when all nine skills were rewritten and the seven reference.md files were folded into their SKILL.md. That is 23.99 days by commit timestamps. In that window the repository received 771 commits; 142 of them touched rules files; 21 touched the sibling SKILL.md; 10 touched reference.md itself. Every one of them passed the commit gate. Exporting the tree from the commit just before the rewrite with git archive <commit>^ CLAUDE.md .claude docs/cli-reference.md package.json | tar -x -C <dir> and running the 26-line check there gives:
The same command on the tree from 2026-08-25, the day of the removal, gives the same two lines. The check would have failed the removal commit itself.
Episode 2: 71 days, a command that never existed. On 2026-06-08, when CLAUDE.md was 1,109 lines long, a "Phase 2" plan was written into it: two commands, pnpm fetch-funnel and pnpm diagnose-funnel, to be built when a trigger fired. Neither was ever added to package.json. The lines lived in CLAUDE.md, loaded at the start of every session, until 2026-08-18 when CLAUDE.md was split into skills and rules files and the plan moved to the architecture-decisions document. 71.2 days. This is a different failure from the first one: not a rename that outran the docs, but a plan phrased as an instruction. The check flags both identically, and it should; an agent reading "run pnpm fetch-funnel" cannot tell the difference either.
Episode 3: 3 days, a false positive. On 2026-08-29 a plain-repost command was deleted on the owner's instruction. The footer of CLAUDE.md carried the version note for that change, and the note named the removed command. The check flags it as missing for 3.0 days, until the footer note moved to the changelog. This is the "we deleted this" case from above. If you adopt the check, decide in advance how you handle it. We would rather read one false positive than a suppression list, and the footer convention that caused it no longer exists.
The commit-time test file for our rules files has 28 cases and takes about 24 seconds. Their names (translated from Japanese) group into five kinds:
SKILL.md; every existing skill is reachable from the body's task table.description, between 40 and 1,536 characters, total under a 2,000-character listing budget, name matching the folder, every .md other than scripts/ and assets/; every rules file has a paths: list in its frontmatter.
Notice what is not there. Not one of the 28 cases opens package.json. The reference-integrity cases check that names in CLAUDE.md resolve to skills, and that is real value, but the relation they check is prose-to-prose. Two other test files in the repository do check command names against package.json: one verifies that a list of weekly measurement commands all exist as scripts, and one verifies that a "mechanization" declaration in a report names a real script. Both check names that live in TypeScript or JSON, not in Markdown. So the honest answer to the first reader is: no. The shape checks catch a rule that is too long, mislabeled or unreachable. They catch nothing about whether the words inside a well-shaped rule are still true, and they were passing throughout the 24 days.
This is the second reader's question, and the 24-day episode is the cleanest answer we have, because the rule was fixed by a test, just not by a test about names. The layout case ("no .md other than SKILL.md in a skill folder") was introduced in the 2026-09-18 rewrite. Satisfying it meant deleting or merging seven reference.md files, and merging is when someone finally reread line 165. The stale instruction did not survive because nobody cared; it survived because for 24 days nobody had a reason to open that line, and the commit that removed the script had already edited the same file and believed it was done.
That points at the two ways a name check can fail you after it exists. First, the fix lands in the wrong place: the test goes red, and the shortest path to green is adding the name to an allowlist or deleting the test, not deleting the sentence. The 26-line script prints file:line for every hit for this reason; the cheapest fix should be the edit to the line. Second, the fix is a mass rewrite. All seven of our reference.md files disappeared in one commit that also rewrote every SKILL.md; a stale line can be fixed by accident in such a commit, and a new one can be introduced by accident just as easily. A check that runs on every commit is worth more than a periodic review precisely because rewrites are when references churn.
We did also check the other direction. Nine of the 199 scripts are not mentioned in any of the 14 files. Two are lint and format helpers; the other seven each have their own source file under src/commands/ and no rule that tells the agent when to run them. We have not audited whether each of the seven is intentionally unlisted, so we are not calling them a problem, but "exists in package.json, unreachable from any rule" is the mirror image of the article's subject, and the same script can list it with a three-line change.
The same 14 files contain 242 backtick-quoted paths under src/, test/, state/, docs/, scripts/, content/, prompts/, logs/ and reports/ (134 unique). Checking each against the filesystem with fs.existsSync finds 3 that do not exist: a temporary diagnosis script that a skill tells the agent to create and then delete, and two ledger files that are created by the command that first writes them. All three are correct as written. A path check therefore needs an allowlist from day one, or a convention (we did not have one) that distinguishes "this file exists" from "this file will exist". We are not adding a path check to the gate on the strength of this measurement; a check with three known false positives and zero true positives on day one is a check people learn to ignore.
The name check has a blind spot the size of the problem: a command that still exists but whose input, output or behaviour changed. Our CLI reference documents input and output shapes for 202 commands in prose; none of that is checked against the TypeScript, and we have no measurement of how often it drifts. The history replay only sees names, and only in the files we listed; the 71-day episode was in CLAUDE.md, which is the one file we would have expected to be read most. Commit timestamps give the gap, not the number of sessions that actually tried the missing command; our run logs for that period do not record package-manager failures in a form we could count without writing a parser, so we are not claiming a number there.
If your rules files name commands, the file-shape checks you already have are not checking those names. The 26-line script above does, it is dependency-free, and on our repository it would have caught the one real stale instruction on the day it was created rather than 24 days later. Scope it to the files that load into the agent's context, read the hits, and let it fail the commit. The official advice to review rules files "periodically" is right; it just does not say that the period, left to itself, was 24 days for us.
Rulestack maintains rules files, skills and the checks that keep them honest for Claude Code, available at rulestack.gumroad.com. The 707-reference scan described here runs as a test in our repository, and the same shape check is bundled with our rule packs.
When the scan catches its next stale command name, we will say so on @ai-shop.bsky.social, including how many days it had been wrong.