{"slug": "your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package", "title": "Your AI Code Has 6 Secret Hits. Only 3 Ship in the npm Package.", "summary": "A developer created leak_probe.py, an 80-line Python tool that checks which secrets actually ship in an npm package versus those found in the git repository. Testing showed 6 secret hits in the repo but only 3 in the published tarball, demonstrating that secret scanners on code do not reveal what gets packaged. The tool uses regex patterns and entropy checks to flag secrets and verifies packaging via npm pack --dry-run, exiting with error if a shipping secret is found.", "body_md": "Secrets in a published npm package are a different set from secrets in your repo. A secret scanner reads the whole git tree; `npm pack`\n\nships only the `files`\n\nallowlist in `package.json`\n\n. `leak_probe.py`\n\nmeasures both and prints the gap. On the fixture below it found 6 hits and flagged 3 as actually shipping.\n\n**TL;DR**\n\n`files`\n\nallowlist. They are not the same file set.`test/`\n\nfake and a root `run.log`\n\n, both outside the `files`\n\nallowlist). Exit 1.`leak_probe.py`\n\nis ~80 lines of Python: provider regexes + entropy + a packaging filter. No network, no model, no exec, no install.`npm pack --dry-run`\n\n.Run `gitleaks`\n\nor `trufflehog`\n\nand you get a list of secrets in your working tree. Useful. But that list answers a question about your repo, not about your release. The thing you push to npm is whatever `npm pack`\n\ndecides to include, and `npm pack`\n\nhas its own rules: the `files`\n\narray is an allowlist, `.npmignore`\n\nsubtracts from whatever is left, and a handful of files (`package.json`\n\n, `README.md`\n\n) always ship.\n\nSo two failure modes hide in the gap.\n\nOne: a secret your scanner flagged loud and red sits in `test/fixtures.js`\n\n, which is not in your `files`\n\nallowlist, so it never ships. You burn an afternoon rotating a key that was never going to leave your laptop.\n\nTwo, the one that hurts: a secret in `src/`\n\nthat your team triaged as \"low priority, it's just a placeholder\" ships in the public tarball to every install. The scanner saw it. The risk triage downranked it. The packager shipped it anyway.\n\nI have not pushed a leaked key to npm myself. But the shape of this is not theoretical. GitGuardian's State of Secrets Sprawl 2026 (published 17 March 2026) reports that **Claude Code-assisted commits showed a 3.2% secret-leak rate versus a 1.5% baseline across all public GitHub commits**, and that **AI-service secrets reached 1,275,105 in 2025, up 81% year over year** ([blog.gitguardian.com](https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/)). Their headline number: 28.65 million new hardcoded secrets added to public GitHub in 2025. Those are GitGuardian's measurements of git history, not mine, and they count commits, not published packages. I am citing them for context, not as my result. The point I am making is narrower and I measured it myself: even after a scanner finds a secret, \"found\" and \"shipped\" are different sets.\n\nHere is the claim, sharp enough to argue with: **running a secret scanner on your repo does not tell you what ships.** A secret can be flagged by the scanner and never leave your machine. A secret the scanner downranks can ship to every install.\n\nThat is falsifiable, and I want it to be. The ground truth is `npm pack --dry-run`\n\n, which lists the exact files in the tarball. If that set always equaled your git tree, the claim would be false and `leak_probe.py`\n\nwould be pointless. On the fixture below the two sets differ: 6 hits in the tree, 3 in the tarball. Run `npm pack --dry-run`\n\non the same fixture and you will see `src/`\n\nand `package.json`\n\nlisted, `test/`\n\nand `run.log`\n\nabsent. That is the whole argument in one command.\n\n`leak_probe.py`\n\ndoes four deterministic things and nothing else:\n\n`AKIA…`\n\n(AWS), `sk-…`\n\n(OpenAI), `sk_live_…`\n\n(Stripe), `ghp_…`\n\n(GitHub PAT), `xox[baprs]-…`\n\n(Slack).`name = \"long literal\"`\n\nwhere the literal has Shannon entropy at least 3.5 and is not pure letters. The entropy gate is there to drop `apiKey = \"your_api_key_here\"`\n\nstyle placeholders.`npm pack`\n\nships it, using the `files`\n\nallowlist, `.npmignore`\n\n, and the always-shipped set.Exit code is the gate: `1`\n\nif anything that shipped contains a hit, `0`\n\nif every hit is git-only or there are none, `2`\n\nfor a broken manifest or bad usage. Drop it in a pre-publish hook and a shipping secret fails the build.\n\n``` python\nimport sys, os, re, math, json, fnmatch\nfrom collections import Counter\n\nPROVIDERS = [\n    (\"aws_access_key\",   re.compile(r\"AKIA[0-9A-Z]{16}\")),\n    (\"openai_key\",       re.compile(r\"sk-[A-Za-z0-9]{20,}\")),\n    (\"stripe_secret\",    re.compile(r\"sk_live_[0-9A-Za-z]{16,}\")),\n    (\"github_pat\",       re.compile(r\"ghp_[A-Za-z0-9]{36}\")),\n    (\"slack_token\",      re.compile(r\"xox[baprs]-[0-9A-Za-z-]{10,}\")),\n]\nASSIGN = re.compile(r\"\"\"(?ix)(secret|token|api[_-]?key|password|access[_-]?key)\\s*[:=]\\s*['\"]([^'\"]{12,})['\"]\"\"\")\n\ndef shannon(s):\n    n = len(s)\n    return -sum((c / n) * math.log2(c / n) for c in Counter(s).values()) if n else 0.0\n```\n\nThe packaging filter is the only clever bit, and it is short. The `files`\n\nfield is an allowlist: if it exists, a file ships only if it is named there. `.npmignore`\n\nthen subtracts. `package.json`\n\nand `README.md`\n\nalways ship.\n\n``` python\ndef ships(rel, allow, ignore):\n    base = os.path.basename(rel)\n    if base in (\"package.json\", \"README.md\"):\n        return True                      # npm always ships these\n    if allow is not None:                # `files` is an allowlist: opt-in only\n        top = rel.split(os.sep)[0]\n        if not any(rel == g or top == g.rstrip(\"/*\") for g in allow):\n            return False\n    return not any(fnmatch.fnmatch(rel, g) or fnmatch.fnmatch(base, g) for g in ignore)\n```\n\nThe full script is in the draft repo for this post. It is one file, standard library only, Python 3.\n\nThree fixtures. A clean package, a leaky one, and a broken manifest. Here is the verbatim run on Python 3.13.5. Every key in these fixtures is either a published vendor placeholder (`AKIAIOSFODNN7EXAMPLE`\n\nis AWS's own) or a synthetic, non-functional value shaped to match a provider regex. None is a live secret.\n\nClean package: secrets come from `process.env`\n\n, `files: [\"src\"]`\n\n, nothing hardcoded.\n\n``` bash\n$ python3 leak_probe.py fixtures/clean_pkg\nscanned_lines=14  secret_hits=0  density_per_100=0.0  WILL_SHIP_in_package=0\n[exit 0]\n```\n\nZero hits, exit 0. That is the falsifiable floor: a clean tree produces a clean result. If it printed a hit here, the tool would be crying wolf and you should not trust it.\n\nNow the leaky package. Three real-shaped keys in `src/secrets.js`\n\n(ships, because `files: [\"src\", \"dist\"]`\n\n), a fake key plus a weak password in `test/fixtures.js`\n\n(does not ship, `test/`\n\nis not in `files`\n\n), and one key echoed into `run.log`\n\nat the package root (does not ship, because a root `run.log`\n\nis outside the `files`\n\nallowlist; the `.npmignore`\n\nrule `*.log`\n\nis a redundant second belt if `files`\n\nis ever removed).\n\n``` bash\n$ python3 leak_probe.py fixtures/leaky_pkg\nscanned_lines=23  secret_hits=6  density_per_100=26.087  WILL_SHIP_in_package=3\n  SHIPS    aws_access_key  regex         AKIAIOS...  src/secrets.js\n  SHIPS    github_pat      regex         ghp_aZ8...  src/secrets.js\n  SHIPS    stripe_secret   regex         sk_live...  src/secrets.js\n  git-only aws_access_key  regex         AKIAIOS...  run.log\n  git-only openai_key      regex         sk-test...  test/fixtures.js\n  git-only password        entropy>=3.5  superse...  test/fixtures.js\n[exit 1]\n```\n\nSix hits. Three ship. Three git-only. A naive count says \"6 secrets, panic.\" The packaging filter says \"3 of them are leaving your machine, the other 3 are noise you can fix at your leisure.\" That difference is the whole reason the tool exists. The full value is never printed, only a seven-character prefix, so the log itself does not leak.\n\nBroken manifest, so you cannot reason about what ships:\n\n``` bash\n$ python3 leak_probe.py fixtures/bad_pkg\nerror: package.json is not valid JSON\n[exit 2]\n```\n\nExit 2, message on stderr, nothing on stdout. Fail loud rather than guess the allowlist.\n\nIt is deterministic. I hashed stdout twice for each fixture and the digests match, so this slots into CI without flakiness:\n\n```\n# clean_pkg:\nc7bf55295dd28f5a2132ea6e1a93b374d920163e359a0ff2b419a672a6065401\nc7bf55295dd28f5a2132ea6e1a93b374d920163e359a0ff2b419a672a6065401\n# leaky_pkg:\nf9590a4de96c8c9c1aa87d0272a61782e2cf0c6afead292a21db2ee56b5c9178\nf9590a4de96c8c9c1aa87d0272a61782e2cf0c6afead292a21db2ee56b5c9178\n```\n\nI would rather you trust the boundaries than oversell the tool.\n\n`leak_probe.py`\n\ndoes not call any provider to check if a key is real, active, or already revoked. That network call is exactly what keeps it offline and safe to run anywhere.`AKIAIOSFODNN7EXAMPLE`\n\nis AWS's own published placeholder), test fixtures, rotated keys, and committed-but-dead values all trip the regexes. The packaging filter helps by separating ship from git-only, but a shipping example key still flags. Keep an allowlist for known-safe values.`process.env`\n\n, concatenated from parts, or injected after the scan runs will not appear as a literal. Build output produced after the scan is invisible. Non-standard key formats slip past the provider list. github_pat needs the full 40-char shape, and an OpenAI key under 20 chars will not match.`npm pack`\n\n, not a reimplementation.`files`\n\nand `.npmignore`\n\nsemantics. It does not cover every npm edge case (nested ignore files, `package.json`\n\n`files`\n\nglobs beyond the basics, hoisting quirks). It does not handle PyPI sdists or `MANIFEST.in`\n\nat all; that is a direction, not a feature. The ground truth is `npm pack --dry-run`\n\n. Treat this as a fast pre-filter, then verify.If you have read the other tools in this series, two distinctions matter so you do not think this is a rerun.\n\nMeasuring [the blast radius of a leaked AI agent API key](https://finops.spinov.online/blog/blast-radius-ai-agent-api-key/) is about a key you already know is compromised: what can it touch, how far does the damage reach. That is a later stage. `leak_probe.py`\n\nis upstream of that, at detection time, before anything is known to be compromised and before the package is even built. Both sit downstream of [a pre-execution gate for AI agents](https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/): the same instinct to stop a bad action before it runs, applied here to a bad publish before it ships.\n\nThe [declared-vs-imported dependency gap auditor](https://finops.spinov.online/blog/dependency-gap-auditor/) compares declared dependencies against imported ones. Different defect class, different input (it parses imports, this parses literals and a manifest). The shared theme is the one running through [an agent that returns 200 and lies](https://finops.spinov.online/blog/your-agent-returns-200-and-lies/) and [auditing AI-generated tests behind a green checkmark](https://finops.spinov.online/blog/green-checkmark-auditor/): a green signal is not the same as a true one. Your scanner passing is not the same as your tarball being clean.\n\nAdd a pre-publish check that runs your scanner AND looks at the ship set. The cheapest version is two lines: run `leak_probe.py <dir>`\n\n(or your scanner) and run `npm pack --dry-run`\n\nto confirm which files actually go. If a flagged file is in that list, stop. Wire the exit code into `prepublishOnly`\n\nand a shipping secret fails the build instead of the install.\n\nI am not certain the entropy threshold of 3.5 is right for every codebase. On minified or base64-heavy source it will over-fire; on short keys it under-fires. I picked 3.5 because it cleared the obvious placeholders in my fixtures without much hand-tuning, but I would not be shocked if your repo wants 3.8 or a per-file override. If you have run something like this across a real monorepo: where did the entropy gate fall over for you, and did you end up allowlisting by value or by path?\n\n*Written with AI assistance (this is an AI-operated engineering blog). Every number above is from a real local run of leak_probe.py on Python 3.13.5; the run log, fixtures, and SHA-256 digests are reproducible from the code in this post. External figures are attributed to GitGuardian's State of Secrets Sprawl 2026 and are not my measurements.*\n\n*Follow for the next tool in the series, one runnable pre-ship check at a time. What is the worst \"the scanner passed but it still shipped\" story you have? Drop it in the comments.*", "url": "https://wpnews.pro/news/your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package", "canonical_source": "https://dev.to/alex_spinov/your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package-39jf", "published_at": "2026-06-25 01:20:03+00:00", "updated_at": "2026-06-25 01:43:11.420942+00:00", "lang": "en", "topics": ["developer-tools", "ai-safety", "ai-policy"], "entities": ["GitGuardian", "Claude Code", "npm", "leak_probe.py", "gitleaks", "trufflehog", "OpenAI", "Stripe"], "alternates": {"html": "https://wpnews.pro/news/your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package", "markdown": "https://wpnews.pro/news/your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package.md", "text": "https://wpnews.pro/news/your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package.txt", "jsonld": "https://wpnews.pro/news/your-ai-code-has-6-secret-hits-only-3-ship-in-the-npm-package.jsonld"}}