{"slug": "i-tested-my-static-analysis-tool-against-4-real-open-source-repos-here-s-what-it", "title": "I Tested My Static Analysis Tool Against 4 Real Open-Source Repos. Here's What It Got Wrong.", "summary": "A developer tested their static analysis tool ArchSetu against four real open-source repos (Express.js, stoplightio/prism, liam-hq/liam, gajus/slonik, Hufe921/canvas-editor) and found that most flagged issues were false positives caused by the tool's inability to handle real-world patterns like monorepos, framework conventions, and path aliases. After fixing bugs, the tool correctly identified genuine dead code and undeclared dependencies, leading to 76 passing unit tests.", "body_md": "I've been building [ArchSetu](https://archsetu.com), a static analysis tool that finds dead code, undeclared dependencies, and risky code paths, fully offline, no AI involved, deterministic output every time.\n\nBefore launch, I wanted real proof it worked, not just tests I wrote myself. So I ran it against four real, actively maintained open-source repos, each with roughly 5,000 GitHub stars, and treated every finding as a hypothesis to verify, not a fact to trust.\n\nHere's what happened.\n\nThe very first run, against Express.js, flagged the repo D grade, 85% dead code. That's obviously wrong for a framework millions of projects depend on. Digging in, the bug was in how my tool traced re-exports through index files, it lost track of usage across module boundaries. Once fixed, Express scored a B, 0% false dead code, and I added seven regression tests so it can't silently regress again.\n\nThat one bug set the tone for the rest of this testing round: don't trust a scary number, verify it.\n\nI picked repos with different shapes on purpose: a monorepo API tool, a Next.js app, a plain library, and a single-maintainer editor.\n\n**stoplightio/prism** (API mocking, monorepo, ~5k stars)\n\nDead code: clean, 0 findings.\n\nDependencies: initially flagged 37 packages as undeclared. All false positives, caused by my tool only checking the root `package.json`\n\nin a monorepo instead of the actual sub-package that declared them. After fixing that, the list dropped to 6, and manual verification confirmed 5 genuinely undeclared dependencies. I filed [an issue](https://github.com/stoplightio/prism/issues/2827) with Prism's maintainers for the confirmed ones.\n\n**liam-hq/liam** (ER diagram generator, Next.js, ~5k stars)\n\nDead code: initially 21 findings. Several were Next.js framework-reserved exports, like `generateMetadata`\n\nand `ErrorPage`\n\n, functions the framework calls implicitly by file location, not something a plain call-graph trace would ever see as \"used.\" My tool didn't know these conventions existed. After building a framework-convention registry, false positives dropped from 5 to 0 for this category.\n\nDependencies: a separate bug here too, bare imports like `@/components/Foo`\n\nthat resolve through TypeScript path aliases in `tsconfig.json`\n\n, not real npm packages. My tool was flagging local file references as missing packages. Fixed with proper tsconfig discovery and path resolution.\n\n**gajus/slonik** (PostgreSQL client library, ~5k stars)\n\nDead code: 1 finding, a function only called from inside a `.test.ts`\n\nfile via test-runner discovery, not a direct call-graph reference. My tool correctly marked it \"unsafe to remove,\" but that pointed to a broader gap, test-invoked functions have a different risk profile than production code and deserve stricter defaults.\n\n**Hufe921/canvas-editor** (canvas/SVG editor, single maintainer, ~5k stars)\n\nDead code: 8 findings, all correctly marked \"unsafe to remove.\" Six were test helper/factory functions, same pattern as slonik. The other two lived in a plugin-registration directory, a pattern my tool doesn't fully recognize yet.\n\n`package.json`\n\n, not just the rootAll four ship with 76 passing unit tests, 18 of them new.\n\nWell-maintained real-world repos mostly don't have obvious, confidently-fileable dead code lying around. Most of what looked like findings were actually gaps in my tool's understanding of real-world patterns: monorepos, framework conventions, path aliases, test-runner invocation. Finding and fixing those gaps against real code was worth more than any synthetic test suite I could have written myself.\n\nIf you want to see where this goes next, [ArchSetu](https://archsetu.com) is close to launch. Feedback on false positives, especially against your own weird real-world codebase, is genuinely the most useful thing you could send me right now at [archsetu@gmail.com](mailto:archsetu@gmail.com)", "url": "https://wpnews.pro/news/i-tested-my-static-analysis-tool-against-4-real-open-source-repos-here-s-what-it", "canonical_source": "https://dev.to/creatovatic/i-tested-my-static-analysis-tool-against-4-real-open-source-repos-heres-what-it-got-wrong-2464", "published_at": "2026-07-12 12:06:07+00:00", "updated_at": "2026-07-12 12:44:13.928519+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["ArchSetu", "Express.js", "stoplightio/prism", "liam-hq/liam", "gajus/slonik", "Hufe921/canvas-editor"], "alternates": {"html": "https://wpnews.pro/news/i-tested-my-static-analysis-tool-against-4-real-open-source-repos-here-s-what-it", "markdown": "https://wpnews.pro/news/i-tested-my-static-analysis-tool-against-4-real-open-source-repos-here-s-what-it.md", "text": "https://wpnews.pro/news/i-tested-my-static-analysis-tool-against-4-real-open-source-repos-here-s-what-it.txt", "jsonld": "https://wpnews.pro/news/i-tested-my-static-analysis-tool-against-4-real-open-source-repos-here-s-what-it.jsonld"}}