{"slug": "google-joins-openai-and-anthropic-in-promising-new-scraping-controls", "title": "Google joins OpenAI and Anthropic in promising new scraping controls", "summary": "Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive that lets publishers opt out of content scraping for generative AI training, adopting an IETF proposal written by Anthropic technical lead for standards Kevin Kelley. The directive uses a new user agent, \"__usage_ai-training\", with Disallow and Allow rules so site owners can block AI-training crawlers while permitting specific pages such as press kits; Google will interpret it as a stop signal for its Google-Extended crawler used to train Gemini models. Anthropic already supports the directive, Microsoft will start on February 1, 2027, Google in early 2027, and OpenAI in the coming months, though none of it reverses scraping already completed.", "body_md": "# Google joins OpenAI and Anthropic in promising new scraping controls\n\nThe new robots.txt directive will prevent the companies from scraping content for generative AI training.\n\n[Google](https://developers.google.com/search/blog/2026/10/ai-training-control), Anthropic, [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/) and Microsoft have [committed](https://www.seroundtable.com/google-ai-training-control-42430.html) to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. The companies have decided to adopt a [proposal](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley. \n\nAs OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley's proposal will change that by getting multiple AI companies to recognize and honor a single common signal. Publishers have to set up a new user agent in the robots.txt file that's specifically for AI training, which will look like this:\n\nUser-Agent: __usage_ai-training\n\nDisallow: /\n\nAllow: /press-kit/\n\nThat will block crawlers specifically meant to gather data for AI training. The \"Allow\" section gives website operators the option to allow scraping for AI training for specific parts of the page, such as if there's a press kit on it. But this new user-agent section doesn't prevent crawling altogether or prevent content from showing up on Google search. For instance, Google will specifically interpret it as a stop sign specifically for the \"Google-Extended\" user agent, which crawls websites for materials used to train Gemini models.\n\nAnother thing to note is that this directive cannot undo any past scraping the companies have already done. If they'd crawled a page previously, they already have its content. This will just prevent them from scraping websites or specific pages in the future. Anthropic has already [started supporting](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) the new directive, while [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) said it will start supporting it on February 1, 2027. Google said it will support it in early 2027, as well, while OpenAI pledged to implement the new signal in the coming months.", "url": "https://wpnews.pro/news/google-joins-openai-and-anthropic-in-promising-new-scraping-controls", "canonical_source": "https://www.engadget.com/2303225/google-openai-anthropic-ai-crawling-controls/", "published_at": "2026-11-01 23:39:35+00:00", "updated_at": "2026-11-01 23:50:48.292706+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-policy", "agent-protocols", "generative-ai"], "entities": ["Google", "Anthropic", "OpenAI", "Microsoft", "Kevin Kelley", "IETF", "Google-Extended", "Gemini"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/google-joins-openai-and-anthropic-in-promising-new-scraping-controls", "markdown": "https://wpnews.pro/news/google-joins-openai-and-anthropic-in-promising-new-scraping-controls.md", "text": "https://wpnews.pro/news/google-joins-openai-and-anthropic-in-promising-new-scraping-controls.txt", "jsonld": "https://wpnews.pro/news/google-joins-openai-and-anthropic-in-promising-new-scraping-controls.jsonld"}}