Google, Anthropic and others to support new AI crawling opt-out standard

Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC announced plans to support an upcoming website crawling standard that would let site operators block AI developers from using their content to train models, SiliconANGLE reported on October 30, 2026. The standard builds on the existing robots.txt text file that tells crawlers what content they may access. The commitment matters because it would give publishers a machine-readable opt-out mechanism for AI training data.

Google, Anthropic and others to support new AI crawling opt-out standard SiliconANGLE https://siliconangle.com Google LLC, Microsoft Corp., Anthropic https://www.machinebrief.com/glossary/anthropic PBC and OpenAI https://www.machinebrief.com/glossary/openai Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence https://www.machinebrief.com/glossary/artificial-intelligence developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. … The post Google, Anthropic and others to support new AI crawling opt-out standard https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/ appeared first on SiliconANGLE https://siliconangle.com . Get AI news in your inbox Daily digest of what matters in AI. Key Terms Explained Anthropic https://www.machinebrief.com/glossary/anthropic An AI safety company founded in 2021 by former OpenAI researchers, including Dario and Daniela Amodei. Artificial Intelligence https://www.machinebrief.com/glossary/artificial-intelligence The science of creating machines that can perform tasks requiring human-like intelligence — reasoning, learning, perception, language understanding, and decision-making. OpenAI https://www.machinebrief.com/glossary/openai The AI company behind ChatGPT, GPT-4, DALL-E, and Whisper.