Meta Platforms has faced intense scrutiny after announcing upcoming changes to its terms that, essentially, allow AI crawlers to run wild across business accounts.
Yet, the owner of Facebook and Instagram has been doing a version of that on websites for years, website managers say, with small online businesses often left to deal with the repercussions.
Take California-based artist Chelsea Paulus, who says Meta’s crawlers crashed her website last week.
A bot called Meta-ExternalAgent got stuck in a loop on Paulus’s website’s “remove from cart” function. It created more than a hundred hits per minute, growing to about 550,000 hits over the next few days. Her website’s CPU hit 80% and her host warned that the site could go down at 100%.
Paulus blocked Meta’s bot, but two days later, another bot, this one called Meta-WebIndexer, showed up. It brought her CPU to 100% and crashed her website.
So, why do these types of bots crawl the internet? Meta’s web crawler page explains, “The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users,” Meta further states. “In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI.”
Crawlers allow many online companies to properly function. Companies use AI crawlers for things such as training large language models (LLMs) or being able to quickly locate answers for AI assistants. Meanwhile, regular crawlers might work with other programs like search engines to find information ahead of users’ queries.
But these crawlers shouldn’t work at the expense of the websites they crawl.
“In 10 years I have never received any kind of traffic usage warnings,” says Paulus. “While I don’t believe this was an intentional malicious attack, it clearly showed that there are no guardrails to protect websites like mine if these giant bots go rogue or get stuck in a loop.”
According to Cloudflare, bots made up 55.1% of U.S. web traffic over the last four weeks.
Paulus initially tried to block the crawlers by setting up a firewall and updating her robots.txt file—but neither method immediately worked. The bots only stopped gaining access when her host gave Paulus code for the website’s .htaccess, files that let you change how your website functions.
Each bot needs to be individually blocked, a potentially laborious task.
“The issue with this is that every time a new bot does this, we have to update the code, so it is not a long-term solution,” says Paulus. She’s looking into other potential options for the future.
“Like many companies, Meta uses web crawlers for a number of different purposes, and website managers can decide how our crawlers interact with their sites through tools like robots.txt,” a Meta spokesperson told Fast Company when reached for comment. “We provide publicly-available guidance on ways to manage robots.txt for our crawlers, and we respect the preferences site owners set.”
Robyn Gough, who runs a small digital design business in Australia, has faced similar issues to Paulus.
Between September 30 and October 28, her servers logged 2.45 TB of data delivered—97.6% of which went to Meta’s crawlers. Hits came from Meta-ExternalAgent (2.29 TB) and, after blocking that bot, from Meta-WebIndexer (107 GB), with 94,154 recorded automatic downloads of her digital graphics’ zip files and small PDFs.
The small PDFs stand in place of some of the zip files following an earlier surge of downloads. At the time, Gough worried her files were being pirated, and so she swapped out the paid files.
Gough’s website has struggled to handle the traffic and deliver products to actual customers.
“My server has had errors on most days in October, with spikes in the middle and end of the month,” says Gough. “My host told me the crawler traffic may have contributed to a resource spike that took my sites down for several hours, and customers have contacted me with database errors when they tried to download purchases.”
Gough blocked the Meta-ExternalAgent crawler on October 26 and the Meta-WebIndexer on October 28 by making changes to her site’s robots.txt and .htaccess.
Following its block, each bot tried and was refused access to her site thousands of times. As of October 31, Meta-WebIndexer had been rejected about 9,400 times.
Like Paulus, one bot quickly replaced the other: Meta-WebIndexer jumped from under 100 requests to about 6,700 requests the day after Gough blocked Meta-ExternalAgent.
Paulus and Gough have chosen not to publicly share their websites in hopes of avoiding any more unwanted traffic.
Gough did some digging after facing such significant disruptions. She alleges that the Meta-ExternalAgent crawler first accessed her website in August 2024. However, it wasn’t until November 2025 that the hits significantly grew, reaching over 100,000 downloads nearly every month since March.
Gough had to pay $10.51 to her host in August, after exceeding the 2 TB limit for her virtual private server (VPS) with 2.57 TB.
Paulus isn’t sure if these specific Meta crawlers accessed her website prior to the recent surge.
Paulus counts herself lucky that she comes from a web design background and that her hosting providers were so helpful.
“Most people would not know where to turn if this happened to them,” she says. “Situations like this could force small business owners to keep their websites offline if they reach usage capacity or pay for added service out of their own pockets.”
Gough seconds that sentiment: “Micro businesses like mine have no developer and no security team, and we only learn we need to block a crawler once the bill or the outage arrives.”
The learning curve has been incredibly frustrating.
“I have had ongoing outage issues with my site for some time now, but as a novice website owner with limited resources and limited energy, it is difficult to figure out how to fix those issues without help,” Gough explains. “I had no robots.txt rule against Meta’s crawler, and wasn’t aware of how detrimental it could be if I didn’t have one.”
It’s also taken up time Gough could be using to create.
“I’m a one-person business with health issues that limit my energy, and I have little available time to design in the first place,” she says. “Working out what was going wrong has taken a great deal of already scarce time away from making products.”
In the end, Gough used another AI tool, Claude, to help her figure out the issue and block the crawlers.
Gough and Paulus are just two examples of what appears to be a wider issue.
For example, Jonathan Harborne, creator of the LGBT History Project, also used Claude when Meta’s crawlers nearly crashed his site earlier this year, as Fast Company previously reported. Meta’s AI crawlers, alongside Alibaba’s, forced Harborne to double the size of the website’s server. As Paulus experienced, Harborne claims that changes to the site’s robots.txt file didn’t stop Meta from accessing the site.
Sites like Reddit are also filled with users expressing frustration at Meta’s AI crawlers disrupting their websites. Users with small businesses talk about being charged more after Meta accesses their site 300,000 times in one day and seeing their website slow down due to the extra traffic.
A number of users debate whether blocking Meta’s AI bots is feasible or if it will hurt their advertising through the company.
As the owner of sites like Instagram, Facebook, and Threads, Meta has widespread reach across the internet. It’s not an easy thing to just block the company and be done with it.
The power Meta holds over small businesses—and the company’s disruptive crawlers—is all the more relevant following the company’s planned commercial terms update.
In early October, the company announced new language, allowing Meta to crawl and scrape content produced by business accounts to use for everything from development to improvement of its advertising and business products.
Publishers and creatives expressed outrage at being basically forced to offer their content up or risk losing necessary income streams.
The new terms were set to go into effect on October 30, a date that has been subsequently pushed to November 20.
But exactly when, and in what form, the terms of use will go into effect is now up in the air. Last week, Meta announced it would undergo “a full consultation with industry groups and leaders” after hearing publishers’ concerns.
The vague promise, shared on X by Meta’s vice president of communications Andy Stone, included a note that the consultation will likely last later than November 20, meaning the update will be pushed again.
With that said, Gough and Paulus will both likely stay on Meta’s platforms, even if the new terms are put into place. “Unfortunately, I need Meta to survive as a small business,” Paulus puts it.
Gough is still weighing her choices but echoes the sentiment: “I rely on Facebook and Instagram to reach customers, so I can’t easily leave as doing so will hinder income I rely on.”