cd /news/ai-crawlers/google-anthropic-and-others-to-suppo… · home › topics › ai-crawlers › article
[ARTICLE · art-161320] src=machinebrief.com ↗ pub= topic=ai-crawlers verified=true sentiment=· neutral

Google, Anthropic and others to support new AI crawling opt-out standard

Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC announced plans to support an upcoming website crawling standard that would let site operators block AI developers from using their content to train models, SiliconANGLE reported on October 30, 2026. The standard builds on the existing robots.txt text file that tells crawlers what content they may access. The commitment matters because it would give publishers a machine-readable opt-out mechanism for AI training data.

by read1 min views8 publishedOct 31, 2026
Google, Anthropic and others to support new AI crawling opt-out standard
Image: Machinebrief (auto-discovered)

SiliconANGLE Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […]

The post Google, Anthropic and others to support new AI crawling opt-out standard appeared first on SiliconANGLE.

Get AI news in your inbox

Daily digest of what matters in AI.

Key Terms Explained #

Anthropic An AI safety company founded in 2021 by former OpenAI researchers, including Dario and Daniela Amodei.

Artificial Intelligence The science of creating machines that can perform tasks requiring human-like intelligence — reasoning, learning, perception, language understanding, and decision-making.

OpenAI The AI company behind ChatGPT, GPT-4, DALL-E, and Whisper.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @google llc 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/google-anthropic-and…] indexed:0 read:1min 2026-10-31 · —