{"slug": "your-robots-txt-probably-does-not-block-the-ai-crawler-you-think-it-blocks", "title": "Your robots.txt probably does not block the AI crawler you think it blocks", "summary": "A developer discovered that many robots.txt files fail to block AI crawlers due to a subtlety in RFC 9309, where naming a specific user-agent group disables wildcard rules for that bot. The developer also catalogued 42 projects for AI-enabled ecommerce on Magento, finding an uneven distribution with 11 llms.txt implementations but only 3 crawler policy tools. The analysis highlights the challenges of AI integration in ecommerce and the shifting focus from in-chat checkout to product discovery feeds.", "body_md": "Here is a rule that looks like it blocks OpenAI's crawler:\n\n```\nUser-agent: *\nDisallow: /checkout/\nDisallow: /customer/\n\nUser-agent: GPTBot\nCrawl-delay: 10\n```\n\nIt blocks nothing. GPTBot reads its own group, finds one `Crawl-delay`\n\nline and no `Disallow`\n\n, and crawls everything — including `/checkout/`\n\nand `/customer/`\n\n.\n\nUnder [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), a crawler that matches a specific user-agent group uses **only** that group. It does not inherit from `User-agent: *`\n\n. Naming a bot anywhere in the file switches off the wildcard rules for that bot entirely.\n\nSo the moment you add a named group to be more careful, you become less careful. And the failure is invisible: the file reads as though it says something.\n\nI found this while doing something else, and the something else is the rest of this post.\n\nI spent a few weeks cataloguing every project that helps an ecommerce store be found, read, trusted or transacted with by an AI system — for Magento specifically, because that is the platform I work on. The result is a list of 42 projects: [awesome-magento-aeo](https://github.com/angeo-dev/awesome-magento-aeo), CC0, fork it if it is useful.\n\nThe interesting part is not the list. It is the distribution.\n\n| Category | Projects |\n|---|---|\n`llms.txt` discovery files |\n11 |\n| Specifications | 7 |\n| MCP servers | 6 |\n| Agentic checkout | 5 |\n| Structured data | 4 |\n| Crawler policy | 3 |\n| Product feeds | 3 |\n| Auditing | 3 |\n\n**Eleven implementations of llms.txt.** For a file format proposed in 2024 that is, at heart, a Markdown document at your site root.\n\nThree for crawler policy — the layer where the RFC 9309 trap above lives.\n\nThe ecosystem built the easiest layer eleven times and the hard layers three times each.\n\nMy first read was duplicated effort: everyone shipping the same weekend project. That read is wrong.\n\nGenerating `llms.txt`\n\nfor a real store is not trivial. You hit multi-store layouts, page-builder content that has to be resolved rather than dumped as markup, CMS directives, customer-group pricing, and catalogues large enough that naive generation exhausts memory.\n\nThe eleven implementations differ precisely on those axes — cursor-based pagination and generators for large catalogues, store-scoped entity selection, weighted ranking, blog content feeding the file with IndexNow pinging.\n\nThey look identical from outside and diverge completely at the point where the platform gets hard. That is what a young category looks like: several people solving the same problem and disagreeing about which parts matter.\n\nIn March 2026 OpenAI pulled back from in-chat Instant Checkout. Published figures put live merchants somewhere between about a dozen and around thirty, against the \"over a million\" named at launch six months earlier. The model moved toward product discovery inside ChatGPT with the purchase completing on the merchant's own site.\n\nThe Agentic Commerce Protocol did not disappear — its role shifted from checkout toward feeds, promotions and availability. OpenAI's [developer documentation](https://developers.openai.com/commerce) still describes Instant Checkout for approved partners, so the details are contested. The direction is not.\n\nNot one of the five agentic checkout projects on the list is past a stable 1.0.\n\nIf you were scoping a sprint against in-chat checkout, the feed is the realistic route to being surfaced at all. The checkout integration is where the protocols are heading, not where revenue is this quarter.\n\n`llms.txt`\n\n.`llms.txt`\n\npoints systems at the pages that matter.I maintain 11 of the 42 entries on that list. My own projects are listed last within their sections rather than in alphabetical position, where they would otherwise appear first almost everywhere. I read documentation and READMEs rather than source, so I have not independently audited every implementation, and where I was unsure I marked partial rather than complete.\n\nCorrections are welcome and never need justification — especially corrections to mine.\n\nThe full write-up, with the inclusion criteria and the parts I left out, is [on my site](https://angeo.dev/magento-aeo-ecosystem-what-has-been-built/).", "url": "https://wpnews.pro/news/your-robots-txt-probably-does-not-block-the-ai-crawler-you-think-it-blocks", "canonical_source": "https://dev.to/angeo/your-robotstxt-probably-does-not-block-the-ai-crawler-you-think-it-blocks-5g3n", "published_at": "2026-08-17 19:25:48+00:00", "updated_at": "2026-08-17 19:43:26.892912+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "Magento", "RFC 9309", "Agentic Commerce Protocol", "awesome-magento-aeo", "IndexNow"], "alternates": {"html": "https://wpnews.pro/news/your-robots-txt-probably-does-not-block-the-ai-crawler-you-think-it-blocks", "markdown": "https://wpnews.pro/news/your-robots-txt-probably-does-not-block-the-ai-crawler-you-think-it-blocks.md", "text": "https://wpnews.pro/news/your-robots-txt-probably-does-not-block-the-ai-crawler-you-think-it-blocks.txt", "jsonld": "https://wpnews.pro/news/your-robots-txt-probably-does-not-block-the-ai-crawler-you-think-it-blocks.jsonld"}}