# How AI Search & Shopping Agents Actually Crawl CMS Sites (/llms.txt, Accept: text/markdown & MCP)

> Source: <https://dev.to/flexextdev/how-ai-search-shopping-agents-actually-crawl-cms-sites-llmstxt-accept-textmarkdown-mcp-55go>
> Published: 2026-11-14 14:14:24+00:00

Over the past 30 days on our own extension store infrastructure, we analyzed every verified bot request hitting production. The numbers surprised even us:

`378` verified requests`GPTBot` + `OAI-SearchBot`):` 377` requests (`+ 7` direct human click-throughs from `ChatGPT-User`)` 187` requests`87` requests`ClaudeBot`:` 26` requests
AI answer engines and shopping assistants are no longer an edge experiment — they are already half of your technical crawl budget. Yet when an LLM agent fetches a typical WordPress, Joomla, OpenCart, or Craft CMS page, it receives **300–600 KB of mega-menus, SVGs, CSS classes, and modal markup** before reaching the first paragraph of product specifications.

Here is the 4-layer engineering checklist we use across [FlexExt.dev](https://flexext.dev/) and [FlexExt Store](https://flexext.store/) to make PHP CMS and e-commerce sites 100% readable to AI agents without hurting human UX or full-page caches.

`robots.txt`
Many site owners copied blanket `robots.txt` blocks in 2024 that block *both* model-training scrapers and real-time search citation crawlers (`OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`). If `OAI-SearchBot` is disallowed, ChatGPT Search cannot cite or link to your product pages.

A clean 2026 `robots.txt` policy explicitly allows search/citation agents and points crawlers to both your XML sitemap and `/llms.txt`:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://example.com/sitemap.xml
# AI Discovery Index: https://example.com/llms.txt
```

`/llms.txt` and `/llms-full.txt`
Following the `llms.txt` specification, exposing a concise Markdown map at `https://yourdomain.com/llms.txt` gives AI agents a curated, hallucination-free index of your documentation, pricing tiers, canonical URLs, and APIs in under 2,000 tokens.

Make sure your web server serves it with explicit UTF-8 Markdown headers:

```
<Files "llms.txt">
    ForceType "text/markdown; charset=utf-8"
    Header set Content-Type "text/markdown; charset=utf-8"
</Files>
```

`Accept: text/markdown` Content Negotiation (Without Breaking Page Cache!)
Modern AI agents and RAG pipelines increasingly send `Accept: text/markdown` when fetching URLs. Returning a clean Markdown representation of your product or article (title, SKU, live price, stock availability, specs, and FAQs) cuts token noise by **80–90%**.

Two critical rules when implementing this in PHP (WordPress, Joomla, OpenCart, Craft CMS):

`Vary: Accept`` str_contains($_SERVER['HTTP_ACCEPT'] ?? '', 'text/markdown')` is true.

``` php
$accept = $_SERVER['HTTP_ACCEPT'] ?? '';
if (str_contains($accept, 'text/markdown') || ($_GET['format'] ?? '') === 'markdown') {
    header('Content-Type: text/markdown; charset=utf-8');
    header('Vary: Accept');
    echo $markdownRenderer->renderProduct($product);
    exit;
}
```

`Schema.org` Graphs & Agent Discovery Cards
When shopping assistants compare software or physical products, they cross-check visible copy against structured JSON-LD:

`Product` + `Offer` (`price`, `priceCurrency`, `availability`, `url`, `sku`)` FAQPage` (10–15 real buyer questions and answers)`/.well-known/mcp/server-card.json` and `/.well-known/agent-card.json` for Model Context Protocol (MCP) and Google A2A discovery.
To avoid manually running `curl` checks on every client store, we built a **free, zero-signup technical scanner** that runs 25 live HTTP probes across 6 AI & GEO Readiness pillars:

👉 **Live Scanner:** [https://aichecker.flexext.dev/](https://aichecker.flexext.dev/)

It automatically fingerprints 45+ CMS and e-commerce stacks (WordPress, WooCommerce, Joomla 4/5/6, OpenCart 3/4, Craft CMS, Shopware, PrestaShop, Shopify) and generates a downloadable starter kit (`llms.txt`, `robots.txt`, `server-card.json`).

If you want drop-in native plugins that automate `/llms.txt`, `Accept: text/markdown`, and JSON-LD dynamically from your catalog:

`FlexExt Store`):