Children's author and illustrator Chelsea Paulus spent a week in late October fighting to keep her website online after automated traffic from Meta's AI crawlers consumed her hosting resources and threatened to take her business offline.
The incident #
On October 24, Paulus received a warning from her hosting provider, SiteGround, that she had used 80% of her allocated CPU resources. Her four websites share a single hosting plan. When she checked her security dashboard the following Monday, she saw requests arriving several times a second [1].
"I went to WordFence, and then I saw the continuous lines that kept popping up every second," Paulus said. "In WordFence, you can click on the details so you can see the IP address. I looked it up, and that IP address is from a Meta location in Dallas" [1].
The visitor identified itself as "Meta External Agent," a fact later confirmed by SiteGround operators. According to Meta's developers webpage, a Meta External Agent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly" [1].
Paulus, who worked in tech and web design before becoming a full-time artist last year, said she had seen Facebook bots visit before but never at this volume. "This has never happened before," she said. "It has been an incredibly stressful week, even with some of my technical knowledge to handle it" [1].
Escalation and temporary fixes #
Paulus edited her robots.txt file and put up a firewall, but those measures can take 24 hours to take effect. She contacted her host, who provided code to add to a hidden configuration file—"not something that anybody who's a normal person running a website would want to touch, but that's the only way that we could stop the bots" [1].
The fix was temporary. By mid-week, another Meta bot named "Meta Web Indexer" arrived, pushing her hosting resource usage to 100%. Meta's developer guide defines a "Meta Web Indexer" as a crawler that "navigates the web to improve Meta AI search result quality for users" [1].
As of Monday, Paulus said her website was still struggling. She had removed the webpage link from her Instagram profile to prevent additional human clicks from taking the site offline entirely. She said she doesn't have the means to pay for increased traffic allowance because bots are not customers and do not generate income [1].
Meta's response #
A Meta spokesperson told Business Insider that website managers control bot activity and referred to its "Meta Web Crawlers Developer Documentation" page for details on how each bot is used [1].
"Like many companies, Meta uses web crawlers for a number of different purposes, and website managers can decide how our crawlers interact with their sites through tools like robots.txt," the spokesperson said. "We provide publicly available guidance on ways to manage robots.txt for our crawlers, and we respect the preferences site owners set" [1].
SiteGround did not respond to a request for comment. According to a screenshot of an email exchange between SiteGround and Paulus, the company said its "server-side protection" is already limiting these bot attempts [1].
Broader context: new commercial terms #
The incident comes as Meta prepares to implement updated Commercial Terms effective November 20, 2026. According to reporting by Nieman Lab and analysis by Author Media, the terms authorize Meta—or anyone acting on its behalf—to use "web scrapers, bots, robots, spiders, crawlers, and user agents and create test credentials" to crawl domains and URLs "associated with your products or services" [2].
The terms apply to anyone using Meta for business, including managing a Page, running ads, or using Meta Business Suite. For authors and artists, that means anyone with an Instagram business profile, a Facebook page, or a Meta ad account [2].
Legal experts say the terms could be argued to override robots.txt. Meta keeps saying in press statements that site owners can block its crawlers with robots.txt, but it has never put that promise in the contract. No version of the terms includes language about respecting robots.txt [2].
Guardrails and small business impact #
Paulus said Meta needs guardrails to prevent this from happening. "Imagine if the unregulated AI keeps doing this, how long can platforms like Cloudflare continue to offer free services?" she said. "If this keeps happening, I would assume that small businesses are going to fork out more, either for higher usage or for programs to better protect themselves" [1].
She is exploring Cloudflare's human verification tools, which are currently free. There are no laws that prevent an AI bot from crawling a publicly accessible website [1].
The case illustrates a gap between Meta's public statements about respecting robots.txt and the practical experience of a small business owner whose site was overwhelmed before mitigation measures could take effect. Meta's crawlers have been active for years: Cloudflare measured Meta-ExternalAgent at 0.9% of AI crawling in July 2024 and 7.5% by July 2025, ranking second only to Googlebot in the 30 days ending August 31, 2026 [2].
What remains unclear is whether Meta's crawler behavior in this instance reflects a configuration error, a change in crawl rate policy, or the cumulative effect of multiple bots operating simultaneously. Meta has not publicly addressed the specifics of Paulus's case beyond its general statement.
Sources #
1. [Meta's AI Crawler Nearly Took an Artist's Business Offline](https://www.businessinsider.com/meta-ai-crawler-nearly-took-artist-business-offline-2026-10) (businessinsider.com)
2. [Kindle Unlimited Is Dead: Long Live Kindle+ - Author Media](https://www.authormedia.com/kindle-unlimited-is-dead-long-live-kindle) (authormedia.com)
3. [All Stock News](https://stockanalysis.com/news/all-stocks) (stockanalysis.com)