{"slug": "google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard", "title": "Google, Anthropic and others to support new AI crawling opt-out standard", "summary": "Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC announced plans to support an upcoming website crawling standard that would let site operators block AI developers from using their content to train models, SiliconANGLE reported on October 30, 2026. The standard builds on the existing robots.txt text file that tells crawlers what content they may access. The commitment matters because it would give publishers a machine-readable opt-out mechanism for AI training data.", "body_md": "# Google, Anthropic and others to support new AI crawling opt-out standard\n\n[SiliconANGLE](https://siliconangle.com)\n\nGoogle LLC, Microsoft Corp., [Anthropic](https://www.machinebrief.com/glossary/anthropic) PBC and [OpenAI](https://www.machinebrief.com/glossary/openai) Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block [artificial intelligence](https://www.machinebrief.com/glossary/artificial-intelligence) developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […]\n\nThe post [Google, Anthropic and others to support new AI crawling opt-out standard](https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/) appeared first on [SiliconANGLE](https://siliconangle.com).\n\nGet AI news in your inbox\n\nDaily digest of what matters in AI.\n\n## Key Terms Explained\n\n[Anthropic](https://www.machinebrief.com/glossary/anthropic)\n\nAn AI safety company founded in 2021 by former OpenAI researchers, including Dario and Daniela Amodei.\n\n[Artificial Intelligence](https://www.machinebrief.com/glossary/artificial-intelligence)\n\nThe science of creating machines that can perform tasks requiring human-like intelligence — reasoning, learning, perception, language understanding, and decision-making.\n\n[OpenAI](https://www.machinebrief.com/glossary/openai)\n\nThe AI company behind ChatGPT, GPT-4, DALL-E, and Whisper.", "url": "https://wpnews.pro/news/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard", "canonical_source": "https://www.machinebrief.com/news/google-anthropic-and-others-to-support-new-ai-crawling-opt-o-atqs", "published_at": "2026-10-31 02:00:23+00:00", "updated_at": "2026-10-31 03:18:11.174184+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-policy", "artificial-intelligence", "agent-protocols"], "entities": ["Google LLC", "Microsoft Corp.", "Anthropic PBC", "OpenAI Group PBC", "SiliconANGLE"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard", "markdown": "https://wpnews.pro/news/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard.md", "text": "https://wpnews.pro/news/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard.txt", "jsonld": "https://wpnews.pro/news/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard.jsonld"}}