Meta's AI Crawler Crashing My DB, and What I Did About It Meta Platforms Ireland's AI training crawler, meta-externalagent/1.1, overloaded the D1 database of Usero, a feedback tool built on Cloudflare Workers, causing 4,227 D1_ERROR events over 14 days and consuming 81% of the free Sentry error quota. Developer Will Smith traced the issue to the crawler's headless Chrome prefetching route manifests and unauthenticated endpoints, which generated about 40 distinct URLs daily and triggered multiple D1 queries per request. Smith implemented fixes including a robots.txt block and auth gates, noting real user signups remained unaffected at 2 per day. <- All posts /blog Usero Journal Meta's AI Crawler Crashing My DB, and What I Did About It On Friday I got an email from Sentry to say I'd used 81% of free monthly errors. Usero doesn't have that many users, so I was annoyed and suspicious of bots . The top issue was D1 ERROR: D1 DB is overloaded. Requests queued for too long . 4,227 events in 14 days, about 80% of everything Sentry had caught. It was all one user, one IP, hitting / manifest . That's the endpoint React Router uses to prefetch the route manifest for the links on a page. So who is it? The IP was 2a06:98c0:3600::103 . It looks like a datacenter, so I looked it up and found ASN 13335. That's Cloudflare. Usero runs on Cloudflare Workers, so the "user" Sentry saw was the edge itself, and "one user" meant everyone arriving through it. A Sentry user IP in Cloudflare's range isn't a person. I needed the real client IP, so went looking for logs. I spent an hour trying to get Cloudflare's Log Explorer going. The API token didn't have Logs Read. I couldn't work out which token it was because the dashboard only searches tokens by name. Log explorer costs $1/GB. And you have to enable the dataset per zone, which isn't retroactive, so it had nothing for the days I cared about anyway. And I'm cheap. It turned out I already had everything. Workers Observability keeps an invocation record for every request, and it carries the caller's ASN, Cloudflare's bot score, the verified bot category, the real connecting IP, the user agent and whether a cookie came with it. My earlier queries had only been returning my own console.log lines, which have none of that. The trick was filtering on $workers.event.response.status exists , which drops you into the invocation records instead. 100% of the / manifest hits came from Meta Platforms Ireland ASN 32934 , user agent meta-externalagent/1.1 . That's Meta's AI training crawler, and Cloudflare tags it as an AI Crawler. Its sibling meta-webindexer/1.1 link previews was in there too, along with OpenAI's GPTBot. They come from 57.141.0.x and 2a03:2880::/32 . The crawler runs a real headless Chrome, so React Router happily prefetches the manifest for every nav link on every page it lands on. It was also crawling feedback.willsmithte.com , the hostname Usero lived on before it had a name. So Meta's crawler was the trigger. Was it the cause? Eh not really. That was my bad. First, the marketing layout mints a random anonymous client id and drops it into a session cookie for every logged-out visitor. It's how the Get Started flow personalises before you sign up. There was a "skip this for bots" guard, but it only covered /docs . Second, that id ends up in the nav as the Get Started link: /client