# Your robots.txt probably does not block the AI crawler you think it blocks > Source: > Published: 2026-08-17 19:25:48+00:00 Here is a rule that looks like it blocks OpenAI's crawler: ``` User-agent: * Disallow: /checkout/ Disallow: /customer/ User-agent: GPTBot Crawl-delay: 10 ``` It blocks nothing. GPTBot reads its own group, finds one `Crawl-delay` line and no `Disallow` , and crawls everything — including `/checkout/` and `/customer/` . Under [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), a crawler that matches a specific user-agent group uses **only** that group. It does not inherit from `User-agent: *` . Naming a bot anywhere in the file switches off the wildcard rules for that bot entirely. So the moment you add a named group to be more careful, you become less careful. And the failure is invisible: the file reads as though it says something. I found this while doing something else, and the something else is the rest of this post. I spent a few weeks cataloguing every project that helps an ecommerce store be found, read, trusted or transacted with by an AI system — for Magento specifically, because that is the platform I work on. The result is a list of 42 projects: [awesome-magento-aeo](https://github.com/angeo-dev/awesome-magento-aeo), CC0, fork it if it is useful. The interesting part is not the list. It is the distribution. | Category | Projects | |---|---| `llms.txt` discovery files | 11 | | Specifications | 7 | | MCP servers | 6 | | Agentic checkout | 5 | | Structured data | 4 | | Crawler policy | 3 | | Product feeds | 3 | | Auditing | 3 | **Eleven implementations of llms.txt.** For a file format proposed in 2024 that is, at heart, a Markdown document at your site root. Three for crawler policy — the layer where the RFC 9309 trap above lives. The ecosystem built the easiest layer eleven times and the hard layers three times each. My first read was duplicated effort: everyone shipping the same weekend project. That read is wrong. Generating `llms.txt` for a real store is not trivial. You hit multi-store layouts, page-builder content that has to be resolved rather than dumped as markup, CMS directives, customer-group pricing, and catalogues large enough that naive generation exhausts memory. The eleven implementations differ precisely on those axes — cursor-based pagination and generators for large catalogues, store-scoped entity selection, weighted ranking, blog content feeding the file with IndexNow pinging. They look identical from outside and diverge completely at the point where the platform gets hard. That is what a young category looks like: several people solving the same problem and disagreeing about which parts matter. In March 2026 OpenAI pulled back from in-chat Instant Checkout. Published figures put live merchants somewhere between about a dozen and around thirty, against the "over a million" named at launch six months earlier. The model moved toward product discovery inside ChatGPT with the purchase completing on the merchant's own site. The Agentic Commerce Protocol did not disappear — its role shifted from checkout toward feeds, promotions and availability. OpenAI's [developer documentation](https://developers.openai.com/commerce) still describes Instant Checkout for approved partners, so the details are contested. The direction is not. Not one of the five agentic checkout projects on the list is past a stable 1.0. If you were scoping a sprint against in-chat checkout, the feed is the realistic route to being surfaced at all. The checkout integration is where the protocols are heading, not where revenue is this quarter. `llms.txt` .`llms.txt` points systems at the pages that matter.I maintain 11 of the 42 entries on that list. My own projects are listed last within their sections rather than in alphabetical position, where they would otherwise appear first almost everywhere. I read documentation and READMEs rather than source, so I have not independently audited every implementation, and where I was unsure I marked partial rather than complete. Corrections are welcome and never need justification — especially corrections to mine. The full write-up, with the inclusion criteria and the parts I left out, is [on my site](https://angeo.dev/magento-aeo-ecosystem-what-has-been-built/).