{"slug": "weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly", "title": "‘We’ve Heard You’: Meta Tweaks Controversial Website-Scraping Terms Slightly", "summary": "Meta revised its forthcoming commercial terms preview on Friday to remove the explicit word \"train\" and state that the consolidated crawling terms apply only to its ads and business products, responding to criticism from business customers and media outlets. News/Media Alliance spokesperson Sam Quigley said the new language \"just limits what Meta is allowed to use publisher content for, but not their ability to crawl and get the content in the first place,\" adding that with AI training \"not being expressly ruled out, it's still potentially on the table.\" The preview still refers to data collection by \"automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents)\" and does not explicitly commit Meta to honoring robots.txt.", "body_md": "Late on Friday, Meta made some changes to a forthcoming policy, and acknowledged that the action was a response to criticism from business customers and members of the media.\n\n“We’ve heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we’ve clarified that in the [terms](https://www.facebook.com/legal/commercial_terms_preview) themselves,” says a [Friday statement to the Mercury News](https://www.mercurynews.com/2026/10/16/meta-updates-data-scraping-policy-following-criticism/).\n\nThe policy as laid out prior to the change made clear that businesses who use Meta’s products will be subject to a regime in which Meta can use those businesses’ website data for essentially anything, including AI training. The new terms (a preview of [which is available here](https://www.facebook.com/legal/commercial_terms_preview)) appear to scale down the eye-poppingly huge umbrella under which businesses’ data might have been used. The explicit use of the term “train” is no longer in the terms, and the new document emphasizes “advertising and business products.”\n\nBut—also according to the Mercury News—one of the new policy’s prominent critics, [News/Media Alliance](https://www.newsmediaalliance.org/) spokesperson Sam Quigley, said of the new policy, “It appears as if the new language just limits what Meta is allowed to use publisher content for, but not their ability to crawl and get the content in the first place.”\n\nQuigley added, per the Mercury News, “They may have taken out training from the listed uses, but with it not being expressly ruled out, it’s still potentially on the table.”\n\nQuigley also points out that Meta’s statement says there will be no changes to how it uses web crawlers, but the company doesn’t explicitly lay this out in the new terms of service.\n\nThe question is whether Meta will fully respect the wishes laid out in a site’s robots.txt file—best thought of as a red light or green light for web crawler bots, [respected as a rule of the road on the internet since time immemorial](https://gizmodo.com/how-to-use-robots-txt-to-hide-your-embarrassing-interen-1700418477). If Meta actually plans to respect robots.txt requests, the thinking goes, then it shouldn’t be a problem to make that explicit in its terms of service.\n\nThe ambiguity comes at a sensitive time for robots.txt, since the norms around old-fashioned web crawlers are becoming shaky as companies, emboldened by their AI ambitions, seem to want to redefine what a bot—broadly defined—should be allowed to do online.\n\nThe terms preview refers to data collection by “automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents).” Seeing such verbiage at a time when agents are become more common might not be all that reassuring, even the company in question says it’s not changing its use of web crawlers, which are a 90s invention used in conjunction with search engines.\n\nAgents get their own separate rules, [according to Google](https://www.searchenginejournal.com/google-says-why-it-may-ignore-robots-txt-and-negatively-impact-seo/583475/), and Google says it will respect those. Nonetheless, [a common sentiment expressed online](https://news.ycombinator.com/item?id=44596690) holds that as companies like these roll out agentic products, we should all expect a day to come soon when agents ignore robots.txt and simply do whatever they want. To be clear, nothing of the sort is in the Meta terms of service preview.", "url": "https://wpnews.pro/news/weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly", "canonical_source": "https://gizmodo.com/weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly-2000827318", "published_at": "2026-10-17 17:52:49+00:00", "updated_at": "2026-10-17 18:36:31.293433+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-policy", "ai-ethics"], "entities": ["Meta", "News/Media Alliance", "Sam Quigley", "Mercury News", "Google"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly", "markdown": "https://wpnews.pro/news/weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly.md", "text": "https://wpnews.pro/news/weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly.txt", "jsonld": "https://wpnews.pro/news/weve-heard-you-meta-tweaks-controversial-website-scraping-terms-slightly.jsonld"}}