Meta rewrote its Commercial Terms last week, and buried in the fine print is a clause that should make every developer who runs a website sit up: starting October 30, 2026, any business use of Meta's products — running ads, managing a Facebook Page or Group, a product catalog, Business Suite, even WhatsApp Business — authorizes Meta to crawl websites "associated with" that account and use what it collects to train its products. That includes AI at Meta.
This isn't speculation. Nieman Lab reported it on October 15, The Register and Search Engine Journal confirmed the clause wording, and Meta's own preview page carries the October 30 effective date. There is no opt-out. The terms' own language: if you don't agree, "stop all access or use of our Products for business or commercial purposes."
Here's the part that affects you directly: your robots.txt doesn't override this contract. And separately, a lot of sites are accidentally nuking their own AI-search visibility with sloppy bot rules. Let me untangle both, then give you a checklist.
The new paragraph sits in the "Licenses & Rights You Provide Meta" section:
"You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services..."
And then: Meta "may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products." "Train" appears in this paragraph only — the one about your website. That's the word that matters.
Worth noting for balance: on October 16, Meta spokesperson Andy Stone responded on X, calling summaries of the report false, saying the update is meant to let Meta pull accurate product info for ad and content recommendations — "not about scraping news sites." The contract language, however, is written much more broadly than that explanation. Three media lawyers told Nieman Lab the wording could support an argument that Meta is permitted to scrape business sites for AI training. Read the clause, not the tweet.
Blocking meta-externalagent in robots.txt tells Meta's crawler "please don't." It does not change the contract you accepted by keeping your Page or ads running. Meta says it still honors robots.txt preferences — and Meta's own docs say a rule takes up to 24 hours to apply — but the license grant in the terms doesn't mention robots.txt at all.
Practical translation: if your website's content is what you sell — courses, paywalled articles, proprietary data — a robots.txt line is not a legal strategy. Talk to a lawyer about a written carve-out before October 30. For everyone else, robots.txt remains the practical control for the crawler's behavior, just not a contractual one.
One useful distinction from Meta's own crawler docs: meta-externalagent (the training/data-use crawler) is separate from facebookexternalhit (link previews) and meta-webindexer (what gets cited in Meta AI answers). Blocking only meta-externalagent leaves your link previews and AI citations working. Don't nuke all three out of panic.
Here's the mistake I see everywhere, and the data is brutal. A GeoParrot census of the top 5,000 sites (robots.txt fetched September 2026) found that 238 of the 535 sites blocking GPTBot also block OAI-SearchBot — that's 44.5% of GPTBot-blockers accidentally taking their ChatGPT citations offline while trying to opt out of training.
They're two different crawlers with two different jobs:
A PageLens study of 740 top sites (October 12, 2026) found the same pattern: 15% block GPTBot, and of those, 63% also block ChatGPT's search or browsing agents. Only a minority do the thing the guides actually recommend — block training crawlers, keep answer crawlers open.
The same logic applies to the other vendors: ClaudeBot (Anthropic training) vs answer-time fetching, Google-Extended (Gemini training — Google says it doesn't affect Search rankings), PerplexityBot (citations). Decide per purpose, not per company.
A dev.to-published study of the top 1,000 sites found that 8–11% of sites server-refuse AI crawlers their robots.txt allows — a bot-management rule or firewall closing a door the site owner wanted open. robots.txt states your policy; server logs show what actually happens.
Related: Cloudflare retired its one-click "Block AI Bots" feature in September. One analysis found 118 sites silently lost their GPTBot bans in robots.txt around the change. If you set-and-forget a bot rule months ago, verify it's still doing what you think.
1. Audit what "associated" means for you. The clause covers domains, pages, and URLs "associated with your products or services." Check: domains verified in Business Suite, URLs in your Page's about section and link-in-bio, catalog feed domains, app domains. If you run five sites but only one is linked to Meta tooling, know which one the clause actually reaches.
2. Decide per purpose, per bot. Make a deliberate call on training crawlers (GPTBot, ClaudeBot, Google-Extended, meta-externalagent, CCBot) separately from answer crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, meta-webindexer). "Block everything AI" is the 44.5% mistake.
3. Write explicit per-bot rules. Don't rely on the wildcard group — a specific User-agent block overrides the wildcard for that bot, and explicit rules are auditable later. Example skeleton:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: meta-externalagent
Disallow: /
Adjust which bots get Allow vs Disallow based on your own training-data decision from step 2. And remember: Disallow: / under a named bot does not affect your Google rankings.
4. Check the rest of the stack. CDN bot-management rules, WAF settings, and server configs can contradict robots.txt in both directions. After deploying, sample your logs for the bots you allowed — look for 200s, not 403s or challenges.
5. Re-verify old rules. If you ever toggled a "block AI bots" switch (Cloudflare or otherwise), fetch your own robots.txt today and confirm it's still what you intended.
6. Know what robots.txt can't do. It doesn't remove content from already-trained models. It doesn't stop a page from being indexed (a blocked page can still appear via links, usually without a description). To keep a page out of results, allow crawling and use noindex instead.
7. If your content is the product, escalate. Publishers, course sellers, data businesses: the contractual grant is the real issue, not the crawler. Get legal advice on a written carve-out before October 30 — robots.txt alone doesn't answer the contract.
If I ran a content site with a Facebook Page: I'd keep answer crawlers (OAI-SearchBot, PerplexityBot, meta-webindexer) wide open — AI citations are free distribution and killing them by accident is the most common unforced error in this whole space. I'd disallow the pure training crawlers (GPTBot, ClaudeBot, Google-Extended, meta-externalagent) and re-check in a month, because vendors rename bots constantly. And I'd read the actual Meta terms once myself instead of trusting anyone's summary — including this article's.
The uncomfortable truth: "your site trains AI" stopped being a technical question and became a contract question. The technical part — robots.txt, per-bot rules, log verification — is the easy half. The hard half is deciding what your content is worth to you, before October 30 decides it for you.
Have you checked your robots.txt this month? And here's the one I'm genuinely curious about: if Meta's clause reaches your site, are you blocking meta-externalagent — or leaving it open?
No affiliate links in this one — it's a service post, not a buying guide. Sources: Nieman Lab (Oct 15), The Register, Search Engine Journal, GeoParrot's robots.txt census (Sep 2026), PageLens top-sites study (Oct 12, 2026). Terms effective October 30, 2026; verify against Meta's current preview page before you act.