{"slug": "aeo-isn-t-seo-why-you-should-let-bots-crawl-your-site-in-2026", "title": "AEO Isn't SEO: Why You Should Let Bots Crawl Your Site in 2026", "summary": "A developer argues that in 2026, websites should welcome AI crawlers for Answer Engine Optimization (AEO), not block them. They detail the distinct bots from OpenAI, Anthropic, and Perplexity for training, search indexing, and live fetching, each requiring separate robots.txt directives. The post warns that robots.txt is voluntary and unreliable against non-compliant crawlers like Perplexity's undeclared bots, and recommends server-rendered HTML and structured data over canonical tags for AI consumption.", "body_md": "For twenty years, `robots.txt`\n\nexisted to keep bandwidth-hungry crawlers off your server. In 2026, for anyone who wants their work to actually be *found*, that instinct is backwards.\n\nIf your goal is showing up when someone asks Claude, ChatGPT, or Perplexity a question — not just ranking on page one of Google — you need AI crawlers hitting your pages. That's **AEO: Answer Engine Optimization**, and it runs on a completely different mechanism than classic SEO.\n\nHere's what that actually means in practice, without the hand-waving.\n\nThe biggest mistake I see (and made myself, initially) is treating \"AI crawlers\" as a single monolith you can allow or block with one `User-agent: *`\n\nrule. They're not. Each provider runs *multiple*, functionally distinct bots, and blocking one does nothing to the others:\n\n`GPTBot`\n\n(training), `OAI-SearchBot`\n\n(powers ChatGPT Search results), `ChatGPT-User`\n\n(live fetch when a user clicks a citation)`ClaudeBot`\n\n(training), `Claude-SearchBot`\n\n(search indexing), `Claude-User`\n\n(fetches a page at a user's direct request)`PerplexityBot`\n\n(its search index crawler), `Perplexity-User`\n\n(user-triggered fetches)`Google-Extended`\n\n(opts you in/out of Gemini training specifically — separate from `Googlebot`\n\n, which handles classic Search and isn't affected by this toggle)Blocking only ClaudeBot does not block Claude-SearchBot or Claude-User — each token needs its own directive. The same logic applies across every vendor: training, search-indexing, and live-fetch are three separate decisions, not one.\n\nA concrete consequence: GPTBot scrapes content to train OpenAI's models, while OAI-SearchBot indexes content specifically for ChatGPT Search results — so blocking `GPTBot`\n\nalone doesn't remove you from OpenAI's training pool if you left `OAI-SearchBot`\n\nopen, and vice versa. You have to decide, per bot, which behavior you're actually opting in or out of.\n\nWorth being blunt about: robots.txt was standardized as RFC 9309 in 2022, but it's a voluntary request that well-behaved crawlers honor — it carries no legal force and doesn't technically block access. Real enforcement, if you need it, happens at the server or CDN layer.\n\nMost of the major players do comply. GPTBot, ClaudeBot, and PerplexityBot each identify themselves with a distinct user-agent string and obey robots.txt rules. But not everyone plays fair — Perplexity has been caught running undeclared crawlers that rotate user-agents, IPs, and ASNs specifically to evade no-crawl directives, and Cloudflare's own investigation concluded that robots.txt alone isn't a reliable defense against that behavior if a bot chooses to ignore it. If you actually need a hard block against a non-compliant crawler, that's a WAF/CDN-level rule, not a text file.\n\nI want to flag something I said elsewhere that I'd retract: the idea that a `rel=\"canonical\"`\n\ntag is \"the key\" signal telling AI crawlers \"this is the source of truth.\" That's not documented anywhere I could verify. Canonical tags are a Google/Bing convention for deduping search index entries — there's no confirmed spec showing GPTBot, ClaudeBot, or PerplexityBot parse and weight it the same way for training ingestion or answer attribution. Treat that one as an unverified guess, not a fact to build a strategy around.\n\nBased on current documentation and crawler behavior:\n\n**Explicit per-bot robots.txt entries.** List\n\n`GPTBot`\n\n, `OAI-SearchBot`\n\n, `ChatGPT-User`\n\n, `ClaudeBot`\n\n, `Claude-SearchBot`\n\n, `Claude-User`\n\n, `PerplexityBot`\n\n, `Perplexity-User`\n\n, and `Google-Extended`\n\nindividually. A blanket `Allow: /`\n\nfor `*`\n\ndoesn't give you granular control over training vs. search vs. live-fetch.**Server-rendered, crawlable HTML.** Not every AI crawler executes JavaScript. If your content only appears after client-side rendering, some bots simply won't see it.\n\n**Structured data (schema.org).** This one has actual documented consumption by search and AI systems, unlike the canonical-tag claim above.\n\n** llms.txt.** An emerging,\n\n`sitemap.xml`\n\n. Adoption sits at around 10% of domains as of mid-2026, so it's early — but it's low-effort and zero-risk to add.**Check your CDN layer separately from your robots.txt.** This is the one people miss most. If you're on Cloudflare or a similar service, its bot-management dashboard can silently override what your actual `robots.txt`\n\nfile says. Research on Cloudflare's network found a meaningful share of sites accidentally blocking major AI crawlers at the CDN while their robots.txt says \"allow\" — the two layers have to agree, or your intended policy does nothing.\n\n```\n# Allow AI search / retrieval (what drives citations)\nUser-agent: OAI-SearchBot\nAllow: /\nUser-agent: ChatGPT-User\nAllow: /\nUser-agent: Claude-SearchBot\nAllow: /\nUser-agent: Claude-User\nAllow: /\nUser-agent: PerplexityBot\nAllow: /\nUser-agent: Perplexity-User\nAllow: /\n\n# Training crawlers — allow if you want long-term model familiarity\nUser-agent: GPTBot\nAllow: /\nUser-agent: ClaudeBot\nAllow: /\nUser-agent: Google-Extended\nAllow: /\n\n# Standard search engines, unchanged\nUser-agent: *\nAllow: /\n\nSitemap: https://yourdomain.com/sitemap.xml\n```\n\nAdjust the training-bot section if you have genuinely proprietary content you don't want memorized — that's a legitimate reason to `Disallow`\n\n`GPTBot`\n\n/`ClaudeBot`\n\n/`Google-Extended`\n\nspecifically while still keeping the search/retrieval bots open for citations. But for a personal site, a portfolio, or a blog whose entire point is being read — open the gate.\n\n*If you're auditing your own robots.txt, check your server logs for 403s against OAI-SearchBot, Claude-User, and PerplexityBot specifically — that's the fastest way to catch an accidental block costing you AI-search visibility.*\n\nBen Santora - July 2026", "url": "https://wpnews.pro/news/aeo-isn-t-seo-why-you-should-let-bots-crawl-your-site-in-2026", "canonical_source": "https://dev.to/tekk73/aeo-isnt-seo-why-you-should-let-bots-crawl-your-site-in-2026-15oi", "published_at": "2026-07-10 10:40:36+00:00", "updated_at": "2026-07-10 10:44:42.456314+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-tools", "developer-tools"], "entities": ["OpenAI", "Anthropic", "Perplexity", "Google", "Cloudflare", "GPTBot", "ClaudeBot", "PerplexityBot"], "alternates": {"html": "https://wpnews.pro/news/aeo-isn-t-seo-why-you-should-let-bots-crawl-your-site-in-2026", "markdown": "https://wpnews.pro/news/aeo-isn-t-seo-why-you-should-let-bots-crawl-your-site-in-2026.md", "text": "https://wpnews.pro/news/aeo-isn-t-seo-why-you-should-let-bots-crawl-your-site-in-2026.txt", "jsonld": "https://wpnews.pro/news/aeo-isn-t-seo-why-you-should-let-bots-crawl-your-site-in-2026.jsonld"}}