# Google, Anthropic and others to support new AI crawling opt-out standard > Source: > Published: 2026-10-31 02:00:23+00:00 # Google, Anthropic and others to support new AI crawling opt-out standard [SiliconANGLE](https://siliconangle.com) Google LLC, Microsoft Corp., [Anthropic](https://www.machinebrief.com/glossary/anthropic) PBC and [OpenAI](https://www.machinebrief.com/glossary/openai) Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block [artificial intelligence](https://www.machinebrief.com/glossary/artificial-intelligence) developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […] The post [Google, Anthropic and others to support new AI crawling opt-out standard](https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/) appeared first on [SiliconANGLE](https://siliconangle.com). Get AI news in your inbox Daily digest of what matters in AI. ## Key Terms Explained [Anthropic](https://www.machinebrief.com/glossary/anthropic) An AI safety company founded in 2021 by former OpenAI researchers, including Dario and Daniela Amodei. [Artificial Intelligence](https://www.machinebrief.com/glossary/artificial-intelligence) The science of creating machines that can perform tasks requiring human-like intelligence — reasoning, learning, perception, language understanding, and decision-making. [OpenAI](https://www.machinebrief.com/glossary/openai) The AI company behind ChatGPT, GPT-4, DALL-E, and Whisper.