My code checker was wrong. How I turned its false positives into tests

A developer behind the open-source CI checker rebar ran it against 28 public repositories and found three false positives, including a husky 9 hook-executable rule that misread how husky 9 generates hooks and a hardcoded-secret rule that flagged AWS's published example key pair. Each false positive was fixed narrowly and paired with regression tests that fail on the old code while still catching real violations.

Disclosure: I'm the author of rebar https://github.com/Navesz/rebar , the open-source checker this post is about. It is alpha. rebar fails a CI job when a repository breaks a rule. Examples: formatting nobody checks, a hook git will never run, a credential in a tracked file. A checker like that can fail in one way that matters more than any other: accusing a repository that is doing the right thing. After a couple of those, people stop reading the output. Soon after, they turn it off. So I ran it against 28 public repositories and read what it said. Three of the accusations were wrong, and I could prove it. Each case below follows the same order: the accusation, the evidence it was wrong, the fix, and the test that keeps it fixed. The test has to work both ways. It must fail on the old code, and the real violation must still be caught. rebar told mastra https://github.com/mastra-ai/mastra commit 7e21ff5 : hooks-executable ✗ no execute bit in the index, git ignores them on Linux: .husky/pre-commit — fix with: git update-index --chmod=+x .husky/pre-commit The file really is mode 100644, and on Linux git skips a hook that isn't executable. The accusation sounds right, and husky 8 users run into exactly that. mastra uses husky 9, which works differently. husky 9 https://github.com/typicode/husky/blob/v9.1.7/index.js L14-L22 makes three changes: core.hooksPath at .husky/ ; .husky/ /pre-commit with mode 0o755 ; sh -e .husky/pre-commit . Git never executes the committed file, so its mode doesn't matter. I checked this by running git, not just by reading the husky source. I made a repository with .husky/pre-commit at 100644, cloned it, ran npm install , and ran git commit with GIT TRACE=1 , which prints the file git runs: | | husky 9.1.7 | husky 8.0.3 | |---|---|---| | file git ran | .husky/ /pre-commit generated | .husky/pre-commit committed | | rebar before | ✗ | ✗ | | rebar after | not applicable | ✗ | I ran it on Windows because Docker wouldn't start on my machine. Windows shows which file git runs. That Linux skips a non-executable hook comes from git's documentation, not from this run. The fix exempts the hooks husky 9 manages. It doesn't switch the rule off. The tests: husky-v9 has two sides with the same bytes and the same 100644 mode. The only difference is the husky version the repository declares. The new code passes one side and fails the other. The old code fails both, which is what makes it a regression test. husky-own-hook declares husky 9, but also has a hook that git runs directly tooling/hooks/pre-push at 100644. Declaring husky must not silence that one, and it doesn't. rebar told dotenv https://github.com/motdotla/dotenv commit f6390d1 : hardcoded-secret ✗ 10 credential s in tracked files … deleting the line is not enough, what is in history has to be ROTATED Six of the ten were not credentials: AWS ACCESS KEY ID=AKIAIOSF•••••••••••• , twice. That's the example key pair from AWS's own IAM documentation. PRIVATE KEY in the README whose body is Kh9NV... , twice. It was cut short on purpose. PASSWORD: 'password' , in two tests. The value is just the word. A secret scanner that tells people to rotate AWS's example key loses its audience fast. The fix releases those three shapes and nothing near them. The tests are built so the fix can't turn into a hole: vendor-example : the failing side is the same file, and it even keeps AWS's example secret key. Only the access key ID changes, to one that isn't a published example, and it must still be flagged. A check for "contains EXAMPLE" would let it through. elided-pem : the failing side has an ellipsis too, but between real 64-character key lines. That's what someone does when they paste a real key into docs and cut out the middle. value-is-the-word : the same test fixture and the same key, with a value that isn't the word. This fix nearly shipped wrong. Its first version removed 45 findings from mastra and added 6 , and the total still went down. A count only shows the net change. So for this post I ran the scanner again with full lists on every repository, at the same commits, and compared finding by finding: mypassword ; dotenv's other four findings are .env files used as test fixtures. rebar still flags them, on purpose, because it flags any committed .env whatever is in it. I haven't settled that design question, and I didn't touch it. rebar told e2b https://github.com/e2b-dev/e2b commit ccaf9fc : formatter ✗ script format only rewrites — nothing fails when a file is out of format The format script is prettier --write , which only rewrites. But e2b's lint workflow https://github.com/e2b-dev/e2b/blob/ccaf9fc0ffe6ac39c7ec786af7608ab1de19467b/.github/workflows/lint.yml L88-L98 runs pnpm run format and then fails the job if git status --porcelain isn't empty. Rewriting and then failing on any difference is a formatting check. The fix recognizes that pattern, but only inside a single job. A git diff --exit-code in another job looks at a different checkout. write-then-verify must pass; write-then-verify-other-job splits the same two commands across two jobs, and it must fail; I also ran rebar's GitHub Action on a runner with only this rule. prettier --write . exits 1 and prettier --check . exits 0. Seven of the 28 repositories are where I found these problems, and I wrote the fixes looking at them, so of course they pass there. The other 21 came from a search rule I wrote before measuring anything. Those 21 are the real test. | | 7 chosen, before → after | 21 searched, before → after | |---|---|---| | hooks-executable , repos flagged | 1 → 0 | 4 → 1 | | hardcoded-secret , findings | 1319 → 1202 | 664 → 630 | | formatter , repos flagged | 4 → 2 | 14 → 14 | examples/web ui/ , a package inside the repository, and rebar only looks for husky at the root. Still open. poe lint , which runs ruff format --check ; scripts/ci.js that calls prettier --check ; npm test , which runs standard . That last one is arguable. Ten verdicts changed in total. Every one went from failing to passing or not applicable, and no repository was newly accused. My first script for sorting those formatter results said "13 correct, 1 false positive". It looked for the check only on the workflow line itself, and didn't follow poe lint or node scripts/ci.js : the same blind spot as the rule it was auditing. The version before that read files through the API, swallowed an error, and reported "no check" for cognee. Both are fixed, and both now fail loudly. I mention them because they are the same bug as the rule: a checker that is wrong without saying so. The searched repositories were picked partly by the mcp topic, and rebar has rules aimed at MCP server setups. This audit says nothing about those rules. I didn't classify their findings, and there's no comparison group of projects without MCP. So I can't say whether MCP projects are riskier, or whether those signatures catch a real attack. Answering that would need a study of its own. Three fixed false positives aren't an accuracy rate either. They are three bugs, and each one now has a test that fails on the old code. npx github:Navesz/rebar v0.1.0 . npx -p github:Navesz/rebar v0.1.0 rebar-security . Or in CI, as a GitHub Action https://github.com/marketplace/actions/rebar-repository-ruler-and-merge-gate , pinned to the release commit. A tag can be moved; a commit can't: - uses: Navesz/rebar@9d39f1e3ed4cd3190b4635dccf22d7b815a74d5b v0.1.0 with: ruler: both check · security · both If it flags something that is correct, open a false-positive issue https://github.com/Navesz/rebar/issues/new?template=false-positive.yml with the rule id, the line it printed, and a public commit or a minimal repository. Every case above started as one wrong line of output, and each one is now a test. Versions. Before: rebar 9f78d9b https://github.com/Navesz/rebar/commit/9f78d9b6e02f2d8c22bc5bfb67fbdd23de2ac2bc . After: 58aff2c https://github.com/Navesz/rebar/commit/58aff2ca6b88dc6a0b77b9430313bab5ef009c58 . The v0.1.0 release 9d39f1e has the same rules: only the Action, its workflow and the changelog differ. The fixes landed in 41 https://github.com/Navesz/rebar/pull/41 husky, formatter, ci-gates, env-example and 40 https://github.com/Navesz/rebar/pull/40 secrets . Sample. 28 repositories, fixed before any result was seen and not extended afterwards. mcp , ai-agents or developer-tools ; Measurement. Each repository was cloned at depth 50 and measured with both rulers and both versions: 112 runs, 0 errors. The commit measured for each repository is in the raw data. The secrets comparison ran the scanner again with full lists at those same commits, because the ruler's summary cuts each list at about 12 items. Other verdicts that changed not among the three cases : ci-gates on dotenv, because npm run lint ; pnpm check runs turbo run fmt:check lint typecheck ; env-example on serena, because the variable it flagged is one the repository Full table. Repositories flagged before → after, with findings in parentheses. Heuristic rules are reported but don't fail the job unless you pass --heuristics . Overall, rebar-check exits 1 on 28 of 28 repositories and rebar-security on 24 of 28, both before and after. Most of that comes from house-convention rules, such as editorconfig 20 of 21 searched and ai-coauthorship 12 of 21 . Those rules encode one project's conventions, and these projects never adopted them. That's not a defect on their side. | Ruler | Rule | Kind | 7 chosen: repos findings | 21 searched: repos findings | |---|---|---|---|---| | check | ai-coauthorship | deterministic | 6 → 6 48 → 48 | 12 → 12 740 → 740 | | check | ci | deterministic | 0 → 0 | 1 → 1 1 → 1 | | check | ci-gates | deterministic | 2 → 0 2 → 0 | 4 → 4 4 → 4 | | check | dependabot | deterministic | 4 → 4 4 → 4 | 5 → 5 5 → 5 | | check | editorconfig | deterministic | 4 → 4 4 → 4 | 20 → 20 20 → 20 | | check | env-example | deterministic | 6 → 5 6 → 5 | 17 → 17 17 → 17 | | check | fake-ui | deterministic | 1 → 1 1 → 1 | 1 → 1 1 → 1 | | check | formatter | deterministic | 4 → 2 4 → 2 | 14 → 14 14 → 14 | | check | git-identity | deterministic | 1 → 1 1 → 1 | 9 → 9 9 → 9 | | check | hooks-executable | deterministic | 1 → 0 1 → 0 | 4 → 1 4 → 1 | | check | notice | deterministic | 3 → 3 3 → 3 | 7 → 7 7 → 7 | | check | orphan-schema | deterministic | 0 → 0 | 1 → 1 1 → 1 | | check | phone | deterministic | 0 → 0 | 2 → 2 2 → 2 | | check | production-url | heuristic | 4 → 4 174 → 174 | 14 → 14 664 → 664 | | check | raw-hex | heuristic | 4 → 4 59 → 59 | 8 → 8 200 → 200 | | check | tests | deterministic | 0 → 0 | 2 → 2 2 → 2 | | check | typecheck | deterministic | 3 → 3 3 → 3 | 9 → 9 9 → 9 | | security | agent-bypass-invocation | deterministic | 0 → 0 | 2 → 2 5 → 5 | | security | control-bytes | deterministic | 3 → 3 84 → 84 | 5 → 5 5737 → 5737 | | security | disabled-defense | deterministic | 1 → 1 1 → 1 | 1 → 1 2 → 2 | | security | env-committed | deterministic | 2 → 2 7 → 7 | 0 → 0 | | security | hardcoded-secret | deterministic | 6 → 6 1319 → 1202 | 16 → 15 664 → 630 | | security | hidden-markdown-directive | heuristic | 1 → 1 1 → 1 | 0 → 0 | | security | hidden-unicode | deterministic | 4 → 4 114 → 114 | 10 → 10 508 → 508 | | security | indirect-exec-change | heuristic | 2 → 2 5 → 5 | 1 → 1 1 → 1 | | security | mcp-ansi-escape | heuristic | 1 → 1 1 → 1 | 2 → 2 91 → 91 | | security | mcp-server-launch | deterministic | 2 → 2 2 → 2 | 5 → 5 10 → 10 | | security | mixed-script-token | heuristic | 1 → 1 2 → 2 | 0 → 0 | | security | password-without-kdf | deterministic | 1 → 1 10 → 10 | 4 → 4 11 → 11 | Only the rows the three cases touch were classified by reading. Every other row is a count, not a verdict on whether rebar is right. Raw data: the raw JSON from every run, the commit list, the scripts and the classification files are attached to the v0.1.0 release https://github.com/Navesz/rebar/releases/tag/v0.1.0 .