We scanned 1,868 AI-built apps for production readiness, and audited our scanner A scan of 1,868 public apps built with AI coding tools found that only 23% pass an 80-point production-ready bar, 24% have at least one critical finding, and 15% contain a hardcoded secret or API key, according to PathToShip. The scanner's false-positive rate was initially 42% but was reduced to 25% after fixing over-counting issues. This is the post behind every number we publish about AI-built code. It has two halves, and the second one is the uncomfortable one. First: what we found scanning 1,868 public apps built with AI coding tools. Second: how we audited our own scanner to make sure those numbers are honest, including the part where we discovered we'd been over-counting. The short version:most scanners never publish their false-positive rate. We measured ours on a representative sample of our critical findings, and it was about 42%. We fixed the classes behind it, re-scanned all 1,868 apps, cut the rate to about 25%, and the corrected numbers are below. If you arrived from a headline, this is where it has to defend itself. Definitions are pinned, caveats come before results, and the places our scanner gets things wrong are documented, not buried. What we scanned We targeted roughly 1,900 public GitHub repositories built with AI coding tools. A handful were unreachable deleted, emptied, or timed out , leaving 1,868 analyzed . For tool attribution we use hand-checked labels first repos we identified as built with a specific tool from clear evidence, like the tool's own config files or generated artifacts , and fall back to the scanner's automatic detection only where no label exists. Every per-tool number below uses that one method. Nine tools, each with a healthy sample: Bolt, Lovable, v0, Cursor, Replit, Tempo, Base44, Firebase Studio, and Windsurf per-tool counts in the table below , plus 234 repos no method could attribute. One transparency number before any results: our automatic detector now agrees with the hand-checked labels 92% of the time, up from 76% at our first report, after we fixed a detection bug more on that below . Detecting which tool built an app is genuinely hard, because some editors leave almost no trace, which is why the table leans on hand-checked labels rather than the detector alone. What the scanner checks PathToShip runs 80 automated checks across 7 weighted dimensions: Security 30%, Production readiness 18%, Architecture 12%, Scalability 12%, Infrastructure 12%, Code quality 8%, Cost efficiency 8%. Severity sets the penalty: critical −30, high −14, medium −6, low −3, and info-level findings always carry zero penalty: they're observations, not score movers. Two design choices worth knowing: Empty repos don't win. A dimension with nothing to evaluate scores 50, not 100. The scanner does not reward having nothing to find. The production-ready bar is 80/100 , the same threshold our certification uses: one bar, applied identically to every repo, including our own /blog/our-own-scanner-gave-us-56 . The honest scope statement: this is pattern-based static analysis over a capped download of each repo 200 files / 500KB, which keeps a scan around 30 seconds . It cannot see runtime behavior, billing plans, deployment settings, or how data flows between files. Where that bit us, we say so. Selection bias, stated plainly These are public repos. People who build serious production apps tend to keep them private. Our set almost certainly over-represents hobby projects, demos, and learning exercises, and under-represents AI-assisted apps with real users and real revenue. So every statistic here is "of the repos we scanned." We make no claims about AI-built apps in general, and if you quote one of our numbers that way, you're quoting it wrong. The results Score distribution: mean 68.3, median 71, range 31–95. Most of the set lands in the working-but-gapped middle band: shippable demos, not shippable products. Headline rates, of the 1,868 repos scanned, definitions pinned: 23% pass the 80-point production-ready bar. 24% have at least one critical finding , after the accuracy work described in the next section. The pre-audit scanner said 33% on this same set; hold that thought. 15% ship a hardcoded secret or API key , defined as the union of our "Hardcoded Secret/Password" and "Hardcoded API Key" rules, counted per repo. 35% have an XSS-class finding , defined conservatively as the union of our two dangerouslySetInnerHTML rules a broader definition that also counts raw innerHTML gives 43% . By dimension , the pattern is the whole thesis: production readiness is the weakest mean 56 , then security 65 and architecture 63 ; scalability 80 and cost efficiency 88 are strong. AI tools generate efficient code that isn't ready to operate. What's missing is the secrets handling, the error fallbacks, the logging, and the deployment hygiene. The per-tool table The framing that has to come before the table: this is a snapshot of public repos per tool, not a tool ranking. Different tools attract different users building different things. Treat the rows as observations, nothing more, and read the next section before you take any single number to the bank. | Tool | Repos | Median score | Mean security | % with ≥1 critical | |---|---|---|---|---| | Lovable | 259 | 74 | 68 | 31% | | v0 | 259 | 72 | 67 | 26% | | Base44 | 248 | 73 | 72 | 21% | | Firebase Studio | 245 | 72 | 66 | 21% | | Tempo | 243 | 71 | 68 | 25% | | Replit | 165 | 67 | 54 | 21% | | Bolt | 152 | 72 | 72 | 22% | | Cursor | 137 | 68 | 57 | 22% | | Windsurf | 134 | 71 | 56 | 29% | The honest headline of this table: across tools, the share of repos with at least one critical finding now sits in a 21–31% band, down from the 22–53% spread we first reported Lovable and Windsurf at the top, most others in the low 20s . Our first report led with "some tools are far riskier than others." Most of that spread turned out to be false positives landing unevenly across tools. It wasn't real, and the next section is why we can now say so. The findings you'll actually see The most common findings of medium severity or higher, by share of repos: | Finding | % of repos | |---|---| | No Content Security Policy | 90% | | No privacy policy | 88% | | Oversized files 300+ lines | 74% | | No React error boundaries | 60% | | No linting config | 58% | | Inline styles overuse | 51% | | No cache invalidation | 48% | | No hosting config detected | 45% | None of these is exotic. That's the point: the distance between an AI-built demo and a production app is mostly unglamorous, well-understood work. Can you trust these numbers? We audited the scanner to find out. Here's the part most data studies skip. A scanner's numbers are only as good as its false-positive rate, and nobody publishes theirs. We did. A "critical" finding is the most expensive thing our scanner can say: it's what a builder drops everything to fix. So a false critical is our worst error: it inflates our stats and sends someone chasing a problem that isn't there. We'd killed some false-positive classes before, but "we fixed the ones we noticed" isn't "we measured how many are left." So we measured. We hand-judged critical findings against the actual code, one at a time: confirmed real , false positive fired on something safe , or ambiguous real pattern, arguable severity . We did it in two passes: a discovery pass to find the kinds of false positives, then a representative estimation sample of 57 findings , drawn proportionally so its false-positive share equals the fleet's. The result: about 42% of our critical findings were false positives 95% confidence interval ±12 points . Not a comfortable number to publish. So we did the obvious thing: fixed the classes it pointed at and re-ran. Two rounds of accuracy work cut the critical false-positive rate from ~42% to ~25% . That's why the critical rate above is 24% and not 33%, and why the secret rate is 15% and not 25%. What was actually wrong , named because "we found some bugs" isn't accountability: Placeholder secrets in example files. The scanner read postgresql://