{"slug": "the-biggest-names-in-ai-are-backing-a-new-way-to-opt-out-of-scraping", "title": "The biggest names in AI are backing a new way to opt out of scraping", "summary": "OpenAI, Google, Microsoft and Anthropic will adopt a shared \"policy token\" signal in robots.txt that lets publishers apply AI training opt-out preferences across multiple AI companies instead of setting them individually, as reported earlier by Search Engine Roundtable. The signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months; Microsoft will also offer granular controls to opt out of AI training or AI grounding while keeping sites in Bing search results.", "body_md": "OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as [reported earlier by *Search Engine Roundtable*](https://www.seroundtable.com/google-ai-training-control-42430.html). With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.\n\n# The biggest names in AI are backing a new way to opt out of scraping\n\nThe new control will allow publishers to apply their AI training preferences across multiple crawlers.\n\nThe [shared “policy token” signal](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s [robots.txt file](https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders), which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.\n\nThe announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the [Really Simple Licensing (RSL) standard](https://www.theverge.com/news/841222/rsl-licensing-ai-spec-launch) giving publishers the ability to charge crawlers that visit their sites, and [Cloudflare blocking AI crawlers by default](https://www.theverge.com/news/695501/cloudflare-block-ai-crawlers-default).\n\n[OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/), [Google](https://developers.google.com/search/blog/2026/10/ai-training-control), [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), and [Anthropic](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/]) say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Microsoft, however, [will offer more granular controls](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) that will give publishers the ability to opt out of AI training or [AI grounding](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/grounding/overview), the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences.\n\nWhile Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”\n\n**Follow topics and authors** from this story to see more like this in your personalized homepage feed and to receive email updates.", "url": "https://wpnews.pro/news/the-biggest-names-in-ai-are-backing-a-new-way-to-opt-out-of-scraping", "canonical_source": "https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out", "published_at": "2026-10-30 20:26:10+00:00", "updated_at": "2026-10-30 20:50:19.075115+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-policy", "agent-protocols", "artificial-intelligence"], "entities": ["OpenAI", "Google", "Microsoft", "Anthropic", "Kevin Kelley", "Internet Engineering Task Force", "Bing"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-biggest-names-in-ai-are-backing-a-new-way-to-opt-out-of-scraping", "markdown": "https://wpnews.pro/news/the-biggest-names-in-ai-are-backing-a-new-way-to-opt-out-of-scraping.md", "text": "https://wpnews.pro/news/the-biggest-names-in-ai-are-backing-a-new-way-to-opt-out-of-scraping.txt", "jsonld": "https://wpnews.pro/news/the-biggest-names-in-ai-are-backing-a-new-way-to-opt-out-of-scraping.jsonld"}}