How AI Search & Shopping Agents Actually Crawl CMS Sites (/llms.txt, Accept: text/markdown & MCP)

A developer at FlexExt analyzed 30 days of verified bot traffic on the company's extension-store infrastructure and found AI crawlers accounted for a large share of technical crawl budget, with GPTBot and OAI-SearchBot generating 378 verified requests against 187 from other bots and 87 from ClaudeBot. The team published a four-layer checklist for making PHP CMS and e-commerce sites readable to AI agents — separating search/citation crawlers from training scrapers in robots.txt, serving an /llms.txt index, negotiating Accept: text/markdown responses with a Vary: Accept header, and adding Schema.org JSON-LD plus MCP and A2A discovery cards — and released a free 25-probe scanner that fingerprints 45+ CMS stacks and generates starter configuration files.

Over the past 30 days on our own extension store infrastructure, we analyzed every verified bot request hitting production. The numbers surprised even us: 378 verified requests GPTBot + OAI-SearchBot : 377 requests + 7 direct human click-throughs from ChatGPT-User 187 requests 87 requests ClaudeBot : 26 requests AI answer engines and shopping assistants are no longer an edge experiment — they are already half of your technical crawl budget. Yet when an LLM agent fetches a typical WordPress, Joomla, OpenCart, or Craft CMS page, it receives 300–600 KB of mega-menus, SVGs, CSS classes, and modal markup before reaching the first paragraph of product specifications. Here is the 4-layer engineering checklist we use across FlexExt.dev https://flexext.dev/ and FlexExt Store https://flexext.store/ to make PHP CMS and e-commerce sites 100% readable to AI agents without hurting human UX or full-page caches. robots.txt Many site owners copied blanket robots.txt blocks in 2024 that block both model-training scrapers and real-time search citation crawlers OAI-SearchBot , ChatGPT-User , PerplexityBot . If OAI-SearchBot is disallowed, ChatGPT Search cannot cite or link to your product pages. A clean 2026 robots.txt policy explicitly allows search/citation agents and points crawlers to both your XML sitemap and /llms.txt : User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / Sitemap: https://example.com/sitemap.xml AI Discovery Index: https://example.com/llms.txt /llms.txt and /llms-full.txt Following the llms.txt specification, exposing a concise Markdown map at https://yourdomain.com/llms.txt gives AI agents a curated, hallucination-free index of your documentation, pricing tiers, canonical URLs, and APIs in under 2,000 tokens. Make sure your web server serves it with explicit UTF-8 Markdown headers: <Files "llms.txt" ForceType "text/markdown; charset=utf-8" Header set Content-Type "text/markdown; charset=utf-8" </Files Accept: text/markdown Content Negotiation Without Breaking Page Cache Modern AI agents and RAG pipelines increasingly send Accept: text/markdown when fetching URLs. Returning a clean Markdown representation of your product or article title, SKU, live price, stock availability, specs, and FAQs cuts token noise by 80–90% . Two critical rules when implementing this in PHP WordPress, Joomla, OpenCart, Craft CMS : Vary: Accept str contains $ SERVER 'HTTP ACCEPT' ?? '', 'text/markdown' is true. php $accept = $ SERVER 'HTTP ACCEPT' ?? ''; if str contains $accept, 'text/markdown' || $ GET 'format' ?? '' === 'markdown' { header 'Content-Type: text/markdown; charset=utf-8' ; header 'Vary: Accept' ; echo $markdownRenderer- renderProduct $product ; exit; } Schema.org Graphs & Agent Discovery Cards When shopping assistants compare software or physical products, they cross-check visible copy against structured JSON-LD: Product + Offer price , priceCurrency , availability , url , sku FAQPage 10–15 real buyer questions and answers /.well-known/mcp/server-card.json and /.well-known/agent-card.json for Model Context Protocol MCP and Google A2A discovery. To avoid manually running curl checks on every client store, we built a free, zero-signup technical scanner that runs 25 live HTTP probes across 6 AI & GEO Readiness pillars: 👉 Live Scanner: https://aichecker.flexext.dev/ https://aichecker.flexext.dev/ It automatically fingerprints 45+ CMS and e-commerce stacks WordPress, WooCommerce, Joomla 4/5/6, OpenCart 3/4, Craft CMS, Shopware, PrestaShop, Shopify and generates a downloadable starter kit llms.txt , robots.txt , server-card.json . If you want drop-in native plugins that automate /llms.txt , Accept: text/markdown , and JSON-LD dynamically from your catalog: FlexExt Store :