{"slug": "meta-s-ai-crawler-crashing-my-db-and-what-i-did-about-it", "title": "Meta's AI Crawler Crashing My DB, and What I Did About It", "summary": "Meta Platforms Ireland's AI training crawler, meta-externalagent/1.1, overloaded the D1 database of Usero, a feedback tool built on Cloudflare Workers, causing 4,227 D1_ERROR events over 14 days and consuming 81% of the free Sentry error quota. Developer Will Smith traced the issue to the crawler's headless Chrome prefetching route manifests and unauthenticated endpoints, which generated about 40 distinct URLs daily and triggered multiple D1 queries per request. Smith implemented fixes including a robots.txt block and auth gates, noting real user signups remained unaffected at 2 per day.", "body_md": "[<- All posts](/blog)\n\nUsero Journal\n\n# Meta's AI Crawler Crashing My DB, and What I Did About It\n\nOn Friday I got an email from Sentry to say I'd used **81%** of free monthly errors. Usero doesn't have that many users, so I was annoyed and suspicious (of bots).\n\nThe top issue was `D1_ERROR: D1 DB is overloaded. Requests queued for too long`. 4,227 events in 14 days, about 80% of everything Sentry had caught. It was all one user, one IP, hitting `/__manifest`. That's the endpoint React Router uses to prefetch the route manifest for the links on a page.\n\nSo who is it?\n\nThe IP was `2a06:98c0:3600::103`. It looks like a datacenter, so I looked it up and found ASN 13335. That's Cloudflare. Usero runs on Cloudflare Workers, so the \"user\" Sentry saw was the edge itself, and \"one user\" meant everyone arriving through it. A Sentry user IP in Cloudflare's range isn't a person.\n\nI needed the real client IP, so went looking for logs. I spent an hour trying to get Cloudflare's Log Explorer going. The API token didn't have Logs Read. I couldn't work out which token it was because the dashboard only searches tokens by name. Log explorer costs $1/GB. And you have to enable the dataset per zone, which isn't retroactive, so it had nothing for the days I cared about anyway. And I'm cheap.\n\nIt turned out I already had everything. Workers Observability keeps an invocation record for every request, and it carries the caller's ASN, Cloudflare's bot score, the verified bot category, the real connecting IP, the user agent and whether a cookie came with it. My earlier queries had only been returning my own console.log lines, which have none of that. The trick was filtering on `$workers.event.response.status exists`, which drops you into the invocation records instead.\n\n**100%** of the `/__manifest` hits came from Meta Platforms Ireland (ASN 32934), user agent `meta-externalagent/1.1`. That's Meta's AI training crawler, and Cloudflare tags it as an AI Crawler. Its sibling `meta-webindexer/1.1` (link previews) was in there too, along with OpenAI's GPTBot. They come from `57.141.0.x` and `2a03:2880::/32`. The crawler runs a real headless Chrome, so React Router happily prefetches the manifest for every nav link on every page it lands on. It was also crawling `feedback.willsmithte.com`, the hostname Usero lived on before it had a name.\n\nSo Meta's crawler was the trigger. Was it the cause? Eh not really. That was my bad.\n\nFirst, the marketing layout mints a random anonymous client id and drops it into a session cookie for every logged-out visitor. It's how the Get Started flow personalises before you sign up. There was a \"skip this for bots\" guard, but it only covered `/docs`.\n\nSecond, that id ends up in the nav as the Get Started link: `/client_<id>/integrations`. The crawler throws away cookies between pages, so gets a fresh id on every page, and follows every one, meaning ~40 distinct `/client_.../integrations` URLs a day.\n\nThird, the integrations page had no auth gate. Anyone with any id got about **5** D1 queries (does this client exist, is Slack connected, how many app reviews, how much feedback, then the client lookup again) before the page 404'd.\n\nFourth, custom domains. Customers can point `feedback.acme.com` at their public board, so any request to a hostname that isn't usero.io did a D1 lookup to find the owner. It did that before checking whether the path was `/`. `feedback.willsmithte.com` (the old domain) wasn't in my list of app hosts, so every asset, manifest and data request on it paid a query as well.\n\nAny customer impact? The overload came in bursts of a few hours on the mornings of Sep 3, 4 and 5. Sentry showed 63 users failing on `/signup`, which had me worried, but every one of them was a Meta IP. Real signups were still 2/day, same as the rest of the week. So as far as I can tell nobody lost anything, but a person landing in the middle of a burst would have got a slow page or a broken one.\n\nThe obvious fix is a robots.txt line, or a firewall rule, and Meta goes away, but I don't really want to do that. I need the AI overlords to learn about Usero.\n\nSo the fix has to make a crawler visit cost zero database work, and still leave them a Get Started link to index. Bots get no anonymous id. Bots see `/integrations` with no id in it, and a person who clicks that from a search result gets redirected into the normal personalised flow. The integrations page turns strangers away before it runs a query. And the custom domain lookup checks the path before it checks the database.\n\nI've now shipped that. Bot detection now trusts Cloudflare's own verdict (`request.cf.verifiedBotCategory`), with a user agent fallback for meta-externalagent, meta-webindexer, GPTBot, Googlebot and the usual suspects. A bot on any marketing route gets no anonymous id and no Set-Cookie. The custom domain resolver checks the method and path first, so only a GET on `/` of a non-app host ever reaches the database, and willsmithte.com is in the app-host list now.\n\nThe URLs changed too. Bots see a static, indexable `/integrations` page (canonical `https://usero.io/integrations`, a link to `/signup`), with link prefetch turned off so React Router doesn't fire `/__manifest` for it. A logged-out stranger asking for someone else's `/client_<id>/integrations` gets a 404 with `X-Robots-Tag: noindex` before any query runs, and anonymous owners' pages carry noindex too, so a per-session id never ends up in an index. One I found on the way: the landing page A/B test was setting a random `ab_id` cookie on every cookieless crawler hit, so Meta saw a different variant each visit. Bots get the default variant now.\n\nI've also cut most of the per-request console.log lines. Workers Logs are quota-capped, and those lines were pushing out the invocation records that had the answer. In a 24 hour window with something like 200k requests, **39** invocation records survived. That's why my first queries found nothing useful.\n\nMeta's AI crawler was a handy load test, and found some bottlenecks which is nice. Otherwise, my customer traffic is still low enough that D1 is plenty performant-enough. The goal is to scale enough that I'll have to find a cheap postgres to migrate to (that's why I use Prisma, the migration's pretty smooth). But I'll deal with that when I get there.\n\nFor now I'll be monitoring the D1 query count, and hoping I don't reach the Sentry limit in the last 3 weeks of the monthly usage bucket.\n\n## Continue reading\n\n### How to Connect User Feedback to Claude Code and Cursor With a Feedback MCP Server\n\nSix feedback tools now ship MCP servers. Here is what a feedback MCP server gives an agent, how to wire one into Claude Code, Cursor, Claude Desktop, Windsurf or VS Code, and a worked example that ends with the agent asking the tool to open the pull request.\n\n7 min read\n\n### How Frill’s Widget Editor Works: The Preview Is the Production Widget\n\nInside Frill’s widget editor: a preview that boots the production widget with the real key, sub-550ms on every control, and no mobile preview.\n\n8 min read\n\n### How the AnnounceKit Email Digest Works: We Waited for the Draft That Never Came\n\nWe enabled the AnnounceKit email digest, set it to send Saturday, and came back on Saturday. No draft, no digest, and nothing in the product showing whether it ran: no draft view, no send history, and a config that froze into upgrade modals mid-trial while staying armed. Plus the weekly digest we built for Usero the same night: a cadence choice instead of a second channel, a draft-preview email with a one-click skip, and an admin page that always shows the next digest and what happened to the last one. Free.\n\n8 min read\n\n## Build a feedback loop your team actually uses\n\nUsero collects, clusters, and turns user feedback into shipped fixes.\n\n[Get started free](/signup)", "url": "https://wpnews.pro/news/meta-s-ai-crawler-crashing-my-db-and-what-i-did-about-it", "canonical_source": "https://usero.io/blog/meta-ai-crawler-crashing-my-db", "published_at": "2026-09-07 23:20:45+00:00", "updated_at": "2026-09-07 23:31:44.145642+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-infrastructure"], "entities": ["Meta Platforms Ireland", "Usero", "Cloudflare", "Sentry", "D1", "React Router", "Will Smith", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/meta-s-ai-crawler-crashing-my-db-and-what-i-did-about-it", "markdown": "https://wpnews.pro/news/meta-s-ai-crawler-crashing-my-db-and-what-i-did-about-it.md", "text": "https://wpnews.pro/news/meta-s-ai-crawler-crashing-my-db-and-what-i-did-about-it.txt", "jsonld": "https://wpnews.pro/news/meta-s-ai-crawler-crashing-my-db-and-what-i-did-about-it.jsonld"}}