{"slug": "our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for", "title": "Our CLAUDE.md and skills named 190 commands: one pointed at a deleted script for 24 days while 28 shape tests passed", "summary": "A developer built a 26-line shell script, check-script-refs.sh, that cross-checks every `pnpm`, `npm run` and `yarn` command referenced in Markdown rules files against the scripts defined in package.json, exiting non-zero when a referenced name no longer exists. Running it on their own repository (707 references to 190 unique names across CLAUDE.md, three rules files, nine skills and a CLI reference) found zero missing names today, but git history showed one skill file pointed at a deleted command for 24 days while 28 commit-time shape tests passed on 142 commits that touched rules files. The author argues file-shape checks cannot catch a rule that is well-formed but stale, since the agent only fails at runtime with a package manager error.", "body_md": "Our CLAUDE.md, three rules files, nine skills and one CLI reference contain 707 `pnpm <name>` references to 190 unique names, checked against 199 scripts in package.json: today, 0 are missing. Git history tells a less tidy story. One skill file pointed at a deleted command for 24 days while 28 commit-time shape tests passed on 142 commits that touched rules files, 10 of them on that very file.\n\nTwo readers asked variations of the same question after our last piece on commit-time tests for rules files. One: \"Are the file-shape checks enough to catch a rule that reads correctly but no longer matches what the code does?\" The other: \"What happens when the test breaks and the rule stays stale anyway?\" We did not have a measured answer, so this article is the measurement. Everything below was run on our own repository on 2026-09-22 with Claude Code 2.1.278; the numbers are from that run.\n\nOur rules files are dense with commands. CLAUDE.md tells the agent which command to run at the start of a turn, which one records a decision, which one pushes. The skills go further: a skill body is mostly \"run this, read that field, then run this\". A CLI reference file lists every command with its input and output shape. That is exactly the kind of instruction the official memory page recommends (\"Run npm test before committing\" instead of \"Test your changes\"), and it has a cost that the same page names in the paragraph on consistency:\n\n\"Periodically\" is doing a lot of work in that sentence. A rule that says `pnpm append-weekly-analysis` is a perfectly well-formed instruction. It has the right frontmatter, it is under the line limit, it has a description. It just names a script that was deleted three weeks ago. The agent will try it, get a package manager error, and then either improvise or stop. Neither outcome is visible in a shape test.\n\nHere is the whole thing. Drop it in the repo root as `check-script-refs.sh`. It reads `package.json`, walks the Markdown files you point it at (default: `CLAUDE.md`, `.claude/`, `docs/`), pulls every `pnpm x`, `npm run x` and `yarn x`, and prints the names that are not scripts, with file and line. It exits 1 when there is at least one, so it can sit in a pre-commit hook or CI job unchanged.\n\n``` bash\n#!/usr/bin/env bash\n# Cross-check `pnpm <x>` / `npm run <x>` / `yarn <x>` in Markdown against package.json scripts.\n# Usage: bash check-script-refs.sh [files or dirs...]   (default: CLAUDE.md .claude docs)\nset -euo pipefail\ntargets=(\"$@\"); [ $# -eq 0 ] && targets=(CLAUDE.md .claude docs)\nfind \"${targets[@]}\" \\( -name node_modules -o -name worktrees \\) -prune -o -name '*.md' -print 2>/dev/null | sort | node -e \"$(cat <<'JS'\nconst fs = require('fs');\nconst scripts = new Set(Object.keys(JSON.parse(fs.readFileSync('package.json', 'utf8')).scripts || {}));\nconst builtin = new Set(['install', 'add', 'run', 'exec', 'dlx', 'test', 'build', 'start', 'lint', 'i', 'ci']);\nconst re = /\\b(?:pnpm|npm run|yarn)\\s+([A-Za-z][\\w:.-]*)/g;\nlet refs = 0, files = 0; const seen = new Map(), missing = new Map();\nfor (const file of fs.readFileSync(0, 'utf8').split('\\n').filter(Boolean)) {\n  files++;\n  fs.readFileSync(file, 'utf8').split('\\n').forEach((line, i) => {\n    for (const m of line.matchAll(re)) {\n      const name = m[1]; refs++; seen.set(name, (seen.get(name) || 0) + 1);\n      if (!scripts.has(name) && !builtin.has(name)) missing.set(name, [...(missing.get(name) || []), `${file}:${i + 1}`]);\n    }\n  });\n}\nconsole.log(`${files} markdown files, ${refs} script references, ${seen.size} unique names, ${scripts.size} scripts in package.json`);\nfor (const [name, where] of missing) console.log(`MISSING  ${name}  <- ${where.slice(0, 3).join(', ')}${where.length > 3 ? ` (+${where.length - 3})` : ''}`);\nconsole.log(missing.size ? `${missing.size} name(s) not in package.json` : 'ok: every referenced name exists');\nprocess.exit(missing.size ? 1 : 0);\nJS\n)\"\n```\n\nTwenty-six lines. The `builtin` set is the list of package-manager subcommands that are not scripts (`pnpm install`, `pnpm test` and so on); extend it if your docs mention others. The `-prune` on `worktrees` is there because Claude Code's agent worktrees live under `.claude/worktrees/` and contain a full copy of the repository; our first run without it scanned 12,800 Markdown files and reported 25 missing names, 19 of which came only from those snapshots and the product fixture files inside them. Scope the input to files that actually load into the agent's context and the noise disappears.\n\nRun on the files that load into context in our repository today, the output is:\n\n``` bash\n$ bash check-script-refs.sh CLAUDE.md .claude/rules .claude/skills docs/cli-reference.md\n14 markdown files, 707 script references, 190 unique names, 199 scripts in package.json\nok: every referenced name exists\n```\n\nFourteen files: one CLAUDE.md, three files in `.claude/rules/`, nine `SKILL.md` files, one CLI reference. The reference file alone accounts for 202 of the 707 references; the biggest skill has 103. Zero missing today. If we had stopped here, the answer to the first reader would have been \"the shape tests seem fine\", which is the wrong answer.\n\nRun with the default targets, which include the whole `docs/` folder, it reports 6 missing names: an example `pnpm foo` in a changelog, and five names that were removed or never built but are recorded as such in the changelog, an architecture-decisions file and an archived copy of an old CLAUDE.md. Those are history, not instructions, and they are a reminder that the check's hits need a human to read them: the tool cannot tell \"run this\" from \"we deleted this\".\n\nTo find out whether \"zero today\" was luck, we replayed the check across history. The script for that is longer than 30 lines, so here is the method rather than the code: list every commit that touched CLAUDE.md, `.claude/`, the CLI reference or `package.json` (399 commits); at each one, read `package.json` from that commit with `git show <commit>:package.json`, grep the rules files at that commit with `git grep -o -E \"pnpm [a-z][a-z0-9-]*\" <commit> -- CLAUDE.md .claude docs/cli-reference.md`, and record every name that is missing, together with the first commit where it stops being missing. That gives episodes with a start date, an end date and a gap in days. We found 9 episodes; 5 lasted 0.0 days (a documentation commit landed seconds before the `package.json` commit that added the script, an ordering artefact). The other 4 are three real stories.\n\n**Episode 1: 24 days, the real case.** On 2026-08-25 a commit removed a script that appended a cross-theme analysis section to the weekly report, after the owner approved dropping that section. The same commit edited CLAUDE.md, the weekly skill's `SKILL.md`, the CLI reference and the skill's `reference.md`, and added a note at the top of the relevant section saying the section was discontinued. It removed the instruction from `SKILL.md` and from the reference table. It left two lines in `reference.md` untouched: line 165 still said to write two or more themes and record them with `pnpm append-weekly-analysis /tmp/analysis.json`, and line 176 still explained what happens when that command throws. The file was a skill's supporting file, the kind the skills documentation describes as loading only on demand (\"Unlike CLAUDE.md content, a skill's body loads only when it's used, so long reference material costs almost nothing until you need it\"). It was needed every Monday.\n\nThe reference stayed until 2026-09-18, when all nine skills were rewritten and the seven `reference.md` files were folded into their `SKILL.md`. That is 23.99 days by commit timestamps. In that window the repository received 771 commits; 142 of them touched rules files; 21 touched the sibling `SKILL.md`; 10 touched `reference.md` itself. Every one of them passed the commit gate. Exporting the tree from the commit just before the rewrite with `git archive <commit>^ CLAUDE.md .claude docs/cli-reference.md package.json | tar -x -C <dir>` and running the 26-line check there gives:\n\nThe same command on the tree from 2026-08-25, the day of the removal, gives the same two lines. The check would have failed the removal commit itself.\n\n**Episode 2: 71 days, a command that never existed.** On 2026-06-08, when CLAUDE.md was 1,109 lines long, a \"Phase 2\" plan was written into it: two commands, `pnpm fetch-funnel` and `pnpm diagnose-funnel`, to be built when a trigger fired. Neither was ever added to `package.json`. The lines lived in CLAUDE.md, loaded at the start of every session, until 2026-08-18 when CLAUDE.md was split into skills and rules files and the plan moved to the architecture-decisions document. 71.2 days. This is a different failure from the first one: not a rename that outran the docs, but a plan phrased as an instruction. The check flags both identically, and it should; an agent reading \"run `pnpm fetch-funnel`\" cannot tell the difference either.\n\n**Episode 3: 3 days, a false positive.** On 2026-08-29 a plain-repost command was deleted on the owner's instruction. The footer of CLAUDE.md carried the version note for that change, and the note named the removed command. The check flags it as missing for 3.0 days, until the footer note moved to the changelog. This is the \"we deleted this\" case from above. If you adopt the check, decide in advance how you handle it. We would rather read one false positive than a suppression list, and the footer convention that caused it no longer exists.\n\nThe commit-time test file for our rules files has 28 cases and takes about 24 seconds. Their names (translated from Japanese) group into five kinds:\n\n`SKILL.md`; every existing skill is reachable from the body's task table.`description`, between 40 and 1,536 characters, total under a 2,000-character listing budget, `name` matching the folder, every `.md` other than `scripts/` and `assets/`; every rules file has a `paths:` list in its frontmatter.\nNotice what is not there. Not one of the 28 cases opens `package.json`. The reference-integrity cases check that names in CLAUDE.md resolve to skills, and that is real value, but the relation they check is prose-to-prose. Two other test files in the repository do check command names against `package.json`: one verifies that a list of weekly measurement commands all exist as scripts, and one verifies that a \"mechanization\" declaration in a report names a real script. Both check names that live in TypeScript or JSON, not in Markdown. So the honest answer to the first reader is: no. The shape checks catch a rule that is too long, mislabeled or unreachable. They catch nothing about whether the words inside a well-shaped rule are still true, and they were passing throughout the 24 days.\n\nThis is the second reader's question, and the 24-day episode is the cleanest answer we have, because the rule was fixed by a test, just not by a test about names. The layout case (\"no `.md` other than `SKILL.md` in a skill folder\") was introduced in the 2026-09-18 rewrite. Satisfying it meant deleting or merging seven `reference.md` files, and merging is when someone finally reread line 165. The stale instruction did not survive because nobody cared; it survived because for 24 days nobody had a reason to open that line, and the commit that removed the script had already edited the same file and believed it was done.\n\nThat points at the two ways a name check can fail you after it exists. First, the fix lands in the wrong place: the test goes red, and the shortest path to green is adding the name to an allowlist or deleting the test, not deleting the sentence. The 26-line script prints `file:line` for every hit for this reason; the cheapest fix should be the edit to the line. Second, the fix is a mass rewrite. All seven of our `reference.md` files disappeared in one commit that also rewrote every `SKILL.md`; a stale line can be fixed by accident in such a commit, and a new one can be introduced by accident just as easily. A check that runs on every commit is worth more than a periodic review precisely because rewrites are when references churn.\n\nWe did also check the other direction. Nine of the 199 scripts are not mentioned in any of the 14 files. Two are lint and format helpers; the other seven each have their own source file under `src/commands/` and no rule that tells the agent when to run them. We have not audited whether each of the seven is intentionally unlisted, so we are not calling them a problem, but \"exists in `package.json`, unreachable from any rule\" is the mirror image of the article's subject, and the same script can list it with a three-line change.\n\nThe same 14 files contain 242 backtick-quoted paths under `src/`, `test/`, `state/`, `docs/`, `scripts/`, `content/`, `prompts/`, `logs/` and `reports/` (134 unique). Checking each against the filesystem with `fs.existsSync` finds 3 that do not exist: a temporary diagnosis script that a skill tells the agent to create and then delete, and two ledger files that are created by the command that first writes them. All three are correct as written. A path check therefore needs an allowlist from day one, or a convention (we did not have one) that distinguishes \"this file exists\" from \"this file will exist\". We are not adding a path check to the gate on the strength of this measurement; a check with three known false positives and zero true positives on day one is a check people learn to ignore.\n\nThe name check has a blind spot the size of the problem: a command that still exists but whose input, output or behaviour changed. Our CLI reference documents input and output shapes for 202 commands in prose; none of that is checked against the TypeScript, and we have no measurement of how often it drifts. The history replay only sees names, and only in the files we listed; the 71-day episode was in CLAUDE.md, which is the one file we would have expected to be read most. Commit timestamps give the gap, not the number of sessions that actually tried the missing command; our run logs for that period do not record package-manager failures in a form we could count without writing a parser, so we are not claiming a number there.\n\nIf your rules files name commands, the file-shape checks you already have are not checking those names. The 26-line script above does, it is dependency-free, and on our repository it would have caught the one real stale instruction on the day it was created rather than 24 days later. Scope it to the files that load into the agent's context, read the hits, and let it fail the commit. The official advice to review rules files \"periodically\" is right; it just does not say that the period, left to itself, was 24 days for us.\n\n*Rulestack maintains rules files, skills and the checks that keep them honest for Claude Code, available at [rulestack.gumroad.com](https://rulestack.gumroad.com?utm_source=devto&utm_medium=article&utm_campaign=our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for-24-days-while-28-shape-tests-passed). The 707-reference scan described here runs as a test in our repository, and the same shape check is bundled with our rule packs.*\n\n*When the scan catches its next stale command name, we will say so on [@ai-shop.bsky.social](https://bsky.app/profile/ai-shop.bsky.social), including how many days it had been wrong.*", "url": "https://wpnews.pro/news/our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for", "canonical_source": "https://dev.to/rulestack/our-claudemd-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for-24-days-while-28-1g33", "published_at": "2026-09-25 02:17:00+00:00", "updated_at": "2026-09-25 02:28:59.349854+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops"], "entities": ["Claude Code", "pnpm", "npm", "yarn"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for", "markdown": "https://wpnews.pro/news/our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for.md", "text": "https://wpnews.pro/news/our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for.txt", "jsonld": "https://wpnews.pro/news/our-claude-md-and-skills-named-190-commands-one-pointed-at-a-deleted-script-for.jsonld"}}