Over the past 30 days on our own extension store infrastructure, we analyzed every verified bot request hitting production. The numbers surprised even us:
378 verified requestsGPTBot + OAI-SearchBot): 377 requests (+ 7 direct human click-throughs from ChatGPT-User) 187 requests87 requestsClaudeBot: 26 requests
AI answer engines and shopping assistants are no longer an edge experiment — they are already half of your technical crawl budget. Yet when an LLM agent fetches a typical WordPress, Joomla, OpenCart, or Craft CMS page, it receives 300–600 KB of mega-menus, SVGs, CSS classes, and modal markup before reaching the first paragraph of product specifications.
Here is the 4-layer engineering checklist we use across FlexExt.dev and FlexExt Store to make PHP CMS and e-commerce sites 100% readable to AI agents without hurting human UX or full-page caches.
robots.txt
Many site owners copied blanket robots.txt blocks in 2024 that block both model-training scrapers and real-time search citation crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot). If OAI-SearchBot is disallowed, ChatGPT Search cannot cite or link to your product pages.
A clean 2026 robots.txt policy explicitly allows search/citation agents and points crawlers to both your XML sitemap and /llms.txt:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://example.com/sitemap.xml
/llms.txt and /llms-full.txt
Following the llms.txt specification, exposing a concise Markdown map at https://yourdomain.com/llms.txt gives AI agents a curated, hallucination-free index of your documentation, pricing tiers, canonical URLs, and APIs in under 2,000 tokens.
Make sure your web server serves it with explicit UTF-8 Markdown headers:
<Files "llms.txt">
ForceType "text/markdown; charset=utf-8"
Header set Content-Type "text/markdown; charset=utf-8"
</Files>
Accept: text/markdown Content Negotiation (Without Breaking Page Cache!)
Modern AI agents and RAG pipelines increasingly send Accept: text/markdown when fetching URLs. Returning a clean Markdown representation of your product or article (title, SKU, live price, stock availability, specs, and FAQs) cuts token noise by 80–90%.
Two critical rules when implementing this in PHP (WordPress, Joomla, OpenCart, Craft CMS):
Vary: Accept`` str_contains($_SERVER['HTTP_ACCEPT'] ?? '', 'text/markdown') is true.
$accept = $_SERVER['HTTP_ACCEPT'] ?? '';
if (str_contains($accept, 'text/markdown') || ($_GET['format'] ?? '') === 'markdown') {
header('Content-Type: text/markdown; charset=utf-8');
header('Vary: Accept');
echo $markdownRenderer->renderProduct($product);
exit;
}
Schema.org Graphs & Agent Discovery Cards
When shopping assistants compare software or physical products, they cross-check visible copy against structured JSON-LD:
Product + Offer (price, priceCurrency, availability, url, sku) FAQPage (10–15 real buyer questions and answers)/.well-known/mcp/server-card.json and /.well-known/agent-card.json for Model Context Protocol (MCP) and Google A2A discovery.
To avoid manually running curl checks on every client store, we built a free, zero-signup technical scanner that runs 25 live HTTP probes across 6 AI & GEO Readiness pillars:
👉 Live Scanner: https://aichecker.flexext.dev/
It automatically fingerprints 45+ CMS and e-commerce stacks (WordPress, WooCommerce, Joomla 4/5/6, OpenCart 3/4, Craft CMS, Shopware, PrestaShop, Shopify) and generates a downloadable starter kit (llms.txt, robots.txt, server-card.json).
If you want drop-in native plugins that automate /llms.txt, Accept: text/markdown, and JSON-LD dynamically from your catalog:
FlexExt Store):