{"slug": "how-ai-search-shopping-agents-actually-crawl-cms-sites-llms-txt-accept-text-mcp", "title": "How AI Search & Shopping Agents Actually Crawl CMS Sites (/llms.txt, Accept: text/markdown & MCP)", "summary": "A developer at FlexExt analyzed 30 days of verified bot traffic on the company's extension-store infrastructure and found AI crawlers accounted for a large share of technical crawl budget, with GPTBot and OAI-SearchBot generating 378 verified requests against 187 from other bots and 87 from ClaudeBot. The team published a four-layer checklist for making PHP CMS and e-commerce sites readable to AI agents — separating search/citation crawlers from training scrapers in robots.txt, serving an /llms.txt index, negotiating Accept: text/markdown responses with a Vary: Accept header, and adding Schema.org JSON-LD plus MCP and A2A discovery cards — and released a free 25-probe scanner that fingerprints 45+ CMS stacks and generates starter configuration files.", "body_md": "Over the past 30 days on our own extension store infrastructure, we analyzed every verified bot request hitting production. The numbers surprised even us:\n\n`378` verified requests`GPTBot` + `OAI-SearchBot`):` 377` requests (`+ 7` direct human click-throughs from `ChatGPT-User`)` 187` requests`87` requests`ClaudeBot`:` 26` requests\nAI answer engines and shopping assistants are no longer an edge experiment — they are already half of your technical crawl budget. Yet when an LLM agent fetches a typical WordPress, Joomla, OpenCart, or Craft CMS page, it receives **300–600 KB of mega-menus, SVGs, CSS classes, and modal markup** before reaching the first paragraph of product specifications.\n\nHere is the 4-layer engineering checklist we use across [FlexExt.dev](https://flexext.dev/) and [FlexExt Store](https://flexext.store/) to make PHP CMS and e-commerce sites 100% readable to AI agents without hurting human UX or full-page caches.\n\n`robots.txt`\nMany site owners copied blanket `robots.txt` blocks in 2024 that block *both* model-training scrapers and real-time search citation crawlers (`OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`). If `OAI-SearchBot` is disallowed, ChatGPT Search cannot cite or link to your product pages.\n\nA clean 2026 `robots.txt` policy explicitly allows search/citation agents and points crawlers to both your XML sitemap and `/llms.txt`:\n\n```\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nSitemap: https://example.com/sitemap.xml\n# AI Discovery Index: https://example.com/llms.txt\n```\n\n`/llms.txt` and `/llms-full.txt`\nFollowing the `llms.txt` specification, exposing a concise Markdown map at `https://yourdomain.com/llms.txt` gives AI agents a curated, hallucination-free index of your documentation, pricing tiers, canonical URLs, and APIs in under 2,000 tokens.\n\nMake sure your web server serves it with explicit UTF-8 Markdown headers:\n\n```\n<Files \"llms.txt\">\n    ForceType \"text/markdown; charset=utf-8\"\n    Header set Content-Type \"text/markdown; charset=utf-8\"\n</Files>\n```\n\n`Accept: text/markdown` Content Negotiation (Without Breaking Page Cache!)\nModern AI agents and RAG pipelines increasingly send `Accept: text/markdown` when fetching URLs. Returning a clean Markdown representation of your product or article (title, SKU, live price, stock availability, specs, and FAQs) cuts token noise by **80–90%**.\n\nTwo critical rules when implementing this in PHP (WordPress, Joomla, OpenCart, Craft CMS):\n\n`Vary: Accept`` str_contains($_SERVER['HTTP_ACCEPT'] ?? '', 'text/markdown')` is true.\n\n``` php\n$accept = $_SERVER['HTTP_ACCEPT'] ?? '';\nif (str_contains($accept, 'text/markdown') || ($_GET['format'] ?? '') === 'markdown') {\n    header('Content-Type: text/markdown; charset=utf-8');\n    header('Vary: Accept');\n    echo $markdownRenderer->renderProduct($product);\n    exit;\n}\n```\n\n`Schema.org` Graphs & Agent Discovery Cards\nWhen shopping assistants compare software or physical products, they cross-check visible copy against structured JSON-LD:\n\n`Product` + `Offer` (`price`, `priceCurrency`, `availability`, `url`, `sku`)` FAQPage` (10–15 real buyer questions and answers)`/.well-known/mcp/server-card.json` and `/.well-known/agent-card.json` for Model Context Protocol (MCP) and Google A2A discovery.\nTo avoid manually running `curl` checks on every client store, we built a **free, zero-signup technical scanner** that runs 25 live HTTP probes across 6 AI & GEO Readiness pillars:\n\n👉 **Live Scanner:** [https://aichecker.flexext.dev/](https://aichecker.flexext.dev/)\n\nIt automatically fingerprints 45+ CMS and e-commerce stacks (WordPress, WooCommerce, Joomla 4/5/6, OpenCart 3/4, Craft CMS, Shopware, PrestaShop, Shopify) and generates a downloadable starter kit (`llms.txt`, `robots.txt`, `server-card.json`).\n\nIf you want drop-in native plugins that automate `/llms.txt`, `Accept: text/markdown`, and JSON-LD dynamically from your catalog:\n\n`FlexExt Store`):", "url": "https://wpnews.pro/news/how-ai-search-shopping-agents-actually-crawl-cms-sites-llms-txt-accept-text-mcp", "canonical_source": "https://dev.to/flexextdev/how-ai-search-shopping-agents-actually-crawl-cms-sites-llmstxt-accept-textmarkdown-mcp-55go", "published_at": "2026-11-14 14:14:24+00:00", "updated_at": "2026-11-14 14:21:18.899939+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "agent-protocols", "structured-data"], "entities": ["FlexExt", "FlexExt Store", "GPTBot", "OAI-SearchBot", "ClaudeBot", "ChatGPT-User", "PerplexityBot", "Model Context Protocol"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-ai-search-shopping-agents-actually-crawl-cms-sites-llms-txt-accept-text-mcp", "markdown": "https://wpnews.pro/news/how-ai-search-shopping-agents-actually-crawl-cms-sites-llms-txt-accept-text-mcp.md", "text": "https://wpnews.pro/news/how-ai-search-shopping-agents-actually-crawl-cms-sites-llms-txt-accept-text-mcp.txt", "jsonld": "https://wpnews.pro/news/how-ai-search-shopping-agents-actually-crawl-cms-sites-llms-txt-accept-text-mcp.jsonld"}}