RFC 9309

mentions 3 type Person feed RSS

// recent coverage 3 mentions

01:57

2026-09-08

ai-visibility.lastminutedealshq.com

ai-policy

The robots.txt user-agent lines that do not name what they look like

A scan of robots.txt files across the Tranco top 5,000 sites found that 582 of 2,725 sites (21.4%) contain at least one User-agent value that is not a valid product token under RFC 9309, causing crawl…

19:25

2026-08-17

dev.to

artificial-intelligence

Your robots.txt probably does not block the AI crawler you think it blocks

A developer discovered that many robots.txt files fail to block AI crawlers due to a subtlety in RFC 9309, where naming a specific user-agent group disables wildcard rules for that bot. The developer …

10:12

2026-08-15

dev.to

developer-tools

robots.txt vs llms.txt vs sitemap.xml: what each is for

A developer explains the distinct roles of robots.txt, sitemap.xml, and llms.txt, using metaphors: robots.txt is a fence, sitemap.xml is a map, and llms.txt is a brief. The post clarifies common misco…

// co-occurs with top 8 entities

OpenAI 2 GPTBot 1 sitemaps.org 1 Magento 1 Agentic Commerce Protocol 1 awesome-magento-aeo 1 IndexNow 1 Google 1

// topics top 6 topics

developer tools 2 artificial intelligence 1 ai products 1 ai tools 1 ai infrastructure 1 ai policy 1