# When AI-built apps get risky: data from 3,911 repos

> Source: <https://pathtoship.com/blog/when-ai-built-apps-actually-break>
> Published: 2026-11-19 21:31:32+00:00

**Recommendation:** Prioritize security review by codebase size: apps past 10,000 lines showed 29.9%–59.2% critical-finding rates versus 2.9% under 1,000 lines, so scan larger AI-built apps first.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

Every post in this genre lands on the same sentence: *98% of AI-built apps have
a security flaw, scan yours now.* We sell a scanner, so that conclusion is
convenient for us. This post is about the week we spent trying to break it.

The corpus is now **3,911 public repositories** across nine AI builders — Bolt,
Lovable, Cursor, v0, Replit, Windsurf, Base44, Tempo and Firebase Studio. The
methodology, and the correction where we found our own headline attribution
figure was circular and revised it from 92.8% to 78.3%, is in
[the corpus post](https://pathtoship.com/blog/state-of-ai-built-code). This is the follow-on: two new
findings, and the three readings of them that turned out to be wrong.

## Most of the flaws are in apps nobody uses

If AI-built apps were riddled with exploitable holes, you would expect a wave of incidents. So we pulled GitHub metadata for every repository we have scanned:

| Signal | Share of 3,911 repos | 
|---|---|
| No stars, forks, or open issues | **66.3%** | 
| Not pushed in 90 days | **68.1%** | 
| Median days since last push | **216** | 
| Links a live deployment | 40.3% | 
| Maintained *and* showing some outside signal | **19.7%** (777 repos) | 

Two thirds of the corpus is dormant. A missing row-level-security policy on a portfolio site nobody has touched in eight months is not a security problem — there is no data to leak and nobody to leak it to. Those builders are not negligent. They are correct.

This changes how every "98% are insecure" headline should be read, including ours. The flaws are real and as common as reported. Most of them sit in apps that will never have a user.

**The caveat runs the other way, and it is the more important one.** Stars badly
undercount real products: an app with two hundred paying customers has no reason
to collect GitHub stars, and plenty of working products never get a public repo
at all because the platform hosts them. So public repositories are not a
representative sample of *deployed* AI-built apps. They are a sample of a
different population. We can say this corpus is mostly dormant. We cannot say
the deployed world is fine, and neither can anybody else scanning public repos.

## The finding that survived every attempt to kill it

The share of apps carrying at least one **critical** finding rises with the size
of the app, and it rises monotonically:

| Lines of code | Repos | ≥1 critical | 95% lower bound | ≥1 high | 
|---|---|---|---|---|
| under 1,000 | 451 | **2.9%** | 1.7% | 26.8% | 
| 1,000–2,000 | 281 | 5.7% | 3.5% | 49.1% | 
| 2,000–5,000 | 510 | 17.6% | 14.6% | 63.1% | 
| 5,000–10,000 | 1,045 | 15.2% | 13.2% | 63.4% | 
| 10,000–15,000 | 532 | 29.9% | 26.2% | 81.8% | 
| 15,000–25,000 | 460 | 38.5% | 34.1% | 88.3% | 
| 25,000–40,000 | 259 | 50.6% | 44.5% | 86.1% | 
| over 40,000 | 373 | **59.2%** | 54.2% | 88.2% | 

Three in a hundred under a thousand lines. Nearly six in ten past forty
thousand. Intervals are Wilson rather than normal, because at small *n* with a
proportion near zero or one the normal approximation produces intervals of zero
width, which is nonsense.

We tried three ways to make it go away.

**Is it one platform's scaffold?** No. It holds independently for all nine
builders, with lifts between 2.1× and 6.4× comparing under-10,000-line apps to
over-10,000-line apps. Nine for nine.

**Is it an artifact of a scanner change?** No — and this was the real risk,
because the corpus was mid-migration between two scanner builds while we
measured, which is an excellent way to fool yourself. Split by build, the lift
is 3.5× in one and 3.6× in the other. Same gradient.

**Is it false positives piling up?** Probably not, though this is the one we
cannot fully close. Our scanner over-flags: a hand audit of 374 critical and
high findings this month found more false positives than our earlier published
estimate, and the four most common classes were fixed the same week. We will
publish the post-fix rate when it is measured, not before. But an error rate
that does not climb with app size cannot manufacture a 20× slope — it inflates
every row about equally. Unless false positives concentrate in large apps, which
the post-fix audit will check, the *shape* survives the false-positive problem
even though the *levels* do not.

## What we wanted it to mean, and what it means

The first instinct was to call it a threshold: *AI-built apps cross a risk cliff
at 10,000 lines.* Clean, quotable, and wrong.

If large apps were genuinely sloppier, there would be more findings **per line**.
We measured that. Findings per 10,000 lines *falls* with size, 0.708 to 0.178.
For about an hour that looked like a better story — large apps are actually
cleaner per line.

That was wrong too, for a dull reason. Our scanner aggregates findings per rule:
one entry per rule tripped, with repeats collapsed into the title as `(3x)`. The
most criticals any repository can report is therefore bounded by the rule table
— the maximum we observed was 8, against 57 possible critical rules — while line
counts run past ninety thousand. Per-line density *has* to fall as size grows.
It is an artifact of our own output format and says nothing about the code in
either direction.

What is left is unglamorous and true: **this is exposure, not degradation.** The
more you build, the more likely something slipped through somewhere. Nobody gets
worse at writing code as their app grows. There is simply more of it to get
wrong, and one leaked key is enough.

## What to do with this

If your app is a few hundred lines and nobody uses it, genuinely: don't worry about this. The 2.9% figure is real.

If you are past ten thousand lines and have users, the base rate is about 30% and climbing with every feature. The three findings that dominate that range, in order:

1. **Row-level security never enabled** on Supabase tables. The anon key is in
your client bundle by design. Without RLS, that key reads the whole table.
2. **A secret on the client side of the boundary** — anything under`NEXT_PUBLIC_` or`VITE_` , or hardcoded in a component. It is bundled into
JavaScript you ship to strangers.
3. **API routes with no authorization check.** Attackers do not use your
interface. They call your endpoints directly.

None of that requires a tool. You can check all three by hand in an afternoon.

## What this does not measure

- **Precision:** the scanner over-flags. A 374-finding hand audit this month
found more false positives than we had estimated; the main classes are fixed
and the post-fix rate will be published once measured. Until then, read
absolute rates as upper bounds and the gradient as the finding.
- **Recall:** nothing here says what the scanner*misses* . No coverage claim, no
"catches X% of issues." That needs a ground-truth corpus with planted
vulnerabilities, which we have not built.
- **Users, incidents, or harm:** this measures code. Not one number here is
evidence that anybody was breached.
- **Private repositories:** public only — which, given the dormancy finding, is
probably the less serious half of the population.

Every figure above regenerates from the dated corpus snapshot it cites, and the snapshot carries the scanner fingerprint that produced it. If you want a slice of the dataset for your own writing, ask. We would rather more people published numbers on this than have ours be the only ones — particularly since ours have needed correcting more than once.
