{"slug": "my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same", "title": "My linter kept warning the people who did it right. Three times, in the same direction", "summary": "A developer who maintains a linter for agent configuration files discovered that its rule for detecting undeclared external CLIs was inverted: it flagged authors who properly declared dependencies in list form while ignoring those who omitted the declaration entirely. The same pattern recurred across three releases, where rules meant to catch unverified writes instead flagged conscientious authors who merely mentioned commands like `git push` in prohibitions or permissions. The developer fixed the issues and noted that text-matching rules cannot distinguish use from mention, causing false positives to concentrate among careful authors.", "body_md": "I maintain a linter that reads agent config files — `SKILL.md`\n\n, `AGENTS.md`\n\n, `CLAUDE.md`\n\n— and fails CI when they bake in something that only works on the author's machine. One of its rules says: if you call an external CLI, declare it, or the next person won't have it.\n\nDeclaring it means naming it in frontmatter:\n\n```\nrequires: codex\n```\n\nExcept that anyone with more than one dependency writes the list form, because that's what YAML is for:\n\n```\nrequires:\n  - codex\n  - gemini\n```\n\nMy implementation only read the first shape. So the block list — the normal way, the way you write it the moment you have two of anything — was invisible to the linter, and it warned you for an undeclared CLI that you had, in fact, declared.\n\nRead that back slowly. Authors who ignored the dependency question entirely were never flagged, because they never wrote a `requires:`\n\nkey at all. Authors who sat down and wrote the contract properly got a warning telling them they hadn't. **The rule was inverted with respect to the thing it was trying to encourage.**\n\nI shipped that. It went out in a patch release, and I only found it because a commenter used the phrase \"dependency contract\" and I went to re-read my own implementation of it.\n\nTwo comments on a post of mine turned into new rules. One of them, `unverified-write`\n\n, reports a file that changes external state — `git push`\n\n, `npm publish`\n\n, an `INSERT`\n\n— and never reads that state back anywhere.\n\nBefore publishing, I ran it over 586 real skill files pulled from a public registry, found two false-positive shapes in the data, fixed both, and re-measured. Fire rate 0.7%, and every hit I could check by hand was genuine. I felt good about it.\n\nThen I handed the diff to a different model for a pre-publish read, and it produced this input in about a minute:\n\n```\nNever run `git push --force` from this skill.\n```\n\nThat is a `git push`\n\nin a code span, in a file with no read-back anywhere. My rule flagged it as an unverified write.\n\n`AGENTS.md`\n\nand `CLAUDE.md`\n\nare *full* of that sentence. Writing down \"don't push without asking\" is the single most common act of care in that genre of file. I had built a rule that warns you for pushing, specifically because you wrote down that you must not push.\n\nI fixed it, published, and then noticed the same shape one layer further out. Prohibitions were now excluded, but permissions were not:\n\n```\n- `git push` は明示の指示があるときだけ。\n- Only run `git push` when the user asks.\n- `npm publish` requires approval from a maintainer.\n```\n\nNobody who writes those sentences has an unverified write. They have a policy. Three releases, three variations, all pointing the same way: **the warning finds the author who wrote the rule down and misses the one who never mentioned it.**\n\nA text-matching rule cannot see actions. It sees *mentions*. And mentions of a dangerous operation are not distributed randomly across authors — they concentrate in the files of people who thought about it.\n\nThe careless author's `AGENTS.md`\n\ndoesn't say `git push`\n\nanywhere. There is nothing for the rule to catch. The careful author's file says it three times: once to declare when it's allowed, once to forbid the force variant, once in the actual deploy step. Two of those three are not the thing you're detecting, and both of them are evidence of care.\n\nSo the base rate is against you. Among all the files containing the string you match on, the share written by conscientious authors is much higher than in the population — and every false positive you have is drawn from that pool. The people most likely to read your warning carefully, and most likely to uninstall you over it, are the people you are most likely to be wrong about.\n\nStatic analysis has a name for the underlying distinction — use versus mention — and my older rule already knew it. Its CLI check ignores a bare ``codex``\n\nin prose and only fires on `codex exec build`\n\n, an actual invocation with an argument. I wrote that exclusion two releases earlier, in response to the same class of complaint, and then built a new rule without it.\n\nThe part I want to be honest about: running against 586 real files did not catch any of this.\n\nIt couldn't. Published, downloadable skills are written to be *used*; they say \"run this\" far more often than \"never run this.\" The prohibition shape lives in team-internal `AGENTS.md`\n\nfiles that nobody uploads to a registry. My corpus was real data, and it was the wrong real data — biased, in exactly the direction that hid the failure.\n\nThat's worth separating out, because \"test against real data\" is advice I've given in writing and still believe:\n\nThe second one found in one pass what 586 files had not. It cost one prompt. If your detector will run on files you can't see — and a linter always does — you need both, and you should assume the corpus is the more comfortable of the two.\n\nNot more keywords. The fix was making the use/mention distinction structural, then checking the price:\n\n`never`\n\n, `do not`\n\n, 禁止) isn't a step.`only … when`\n\n, `requires approval`\n\n, 〜のときだけ) isn't a step `git push origin main # main only`\n\n.Then the part that isn't optional: re-run the corpus and prove the exclusions didn't eat the signal. Four true positives before, the same four after, fire rate unchanged at 0.7%. An exclusion you didn't measure is just a rule you deleted with extra steps.\n\nThe same review turned up something unrelated but worse. My check for a remote copy looked like this:\n\n```\n\\b(?:scp|rsync)\\b[^\\n]*\\s\\S+@\\S+:\n```\n\nGreedy fill, then a search for `something@something:`\n\n. On a long line containing many `@`\n\nand no colon, it backtracks quadratically: 77ms at 20k characters, 312ms at 40k, and it keeps going up from there. A minified blob or a base64 payload on one line of somebody's repo is enough.\n\nI had been thinking of the input as \"config files people wrote.\" It isn't. It's **arbitrary text from strangers**, and a linter that hangs is a linter that stops a stranger's CI. The fix was to stop the filler crossing an `@`\n\nso there's nothing to backtrack over; an 80,000-character line is now a test case.\n\nWhen a detector matches on text, ask who says that text most often. If the answer is \"the people being careful about it,\" your false positives are not evenly distributed — they're aimed. And the only reliable way to see it is to hand the rule to something that is actively trying to embarrass you, because your own corpus is made of the cases you already knew about.\n\nThe tools: [carrylint](https://github.com/hyuga611/carrylint) is the linter above; the read-back half it deliberately doesn't attempt is [genchi](https://github.com/hyuga611/genchi).", "url": "https://wpnews.pro/news/my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same", "canonical_source": "https://dev.to/hyuga611/my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same-direction-2j45", "published_at": "2026-08-17 09:53:09+00:00", "updated_at": "2026-08-17 10:13:49.143121+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["codex", "gemini"], "alternates": {"html": "https://wpnews.pro/news/my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same", "markdown": "https://wpnews.pro/news/my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same.md", "text": "https://wpnews.pro/news/my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same.txt", "jsonld": "https://wpnews.pro/news/my-linter-kept-warning-the-people-who-did-it-right-three-times-in-the-same.jsonld"}}