Everyone's writing think-pieces about "optimizing for AI search." I wanted numbers. So I pulled together the 2025-2026 data from Cloudflare, Vercel/MERJ, Ahrefs and the AI providers' own docs. Some of it genuinely changed how I build sites. Here's the reality check, dev to dev.
This is the big one. A Vercel/MERJ analysis of 500M+ crawler fetches found that none of the major dedicated AI crawlers execute JavaScript — not OpenAI's GPTBot
/OAI-SearchBot
/ChatGPT-User
, not Anthropic's ClaudeBot
, not PerplexityBot
, not Meta or ByteDance's Bytespider
. They fetch your raw HTML and leave. They do download JS files (~11.5% of ChatGPT's fetches, ~23.8% of Claude's) — they just never run them.
The exception is Google: its Web Rendering Service is shared across Search and Gemini, so Google-Extended renders JS the same way Googlebot does.
Translation: if your content only exists after client-side hydration, every AI engine except Google sees an empty shell.
curl -A "GPTBot" https://yoursite.com/page | grep "your headline"
If that's empty, you need SSR, SSG, or prerendering.
Cloudflare's data shows AI crawler activity up sharply year over year, with GPTBot's volume up ~305% and reaching ~11.7% of all crawler requests by mid-2025. But roughly 86% of that crawling is for model training, not live search retrieval.
And the referrals? Lopsided doesn't cover it. Measured crawl-to-referral ratios hit ~38,065:1 for Anthropic and ~1,091:1 for OpenAI. They read enormously, cite rarely. AI tools still send under 1% of outbound web traffic today.
By early 2026, roughly 68% of Google searches ended without a click, and pages with an AI Overview above them saw click-through drop by around 60%. Being "on page one" is worth less every quarter; being the cited source is worth more.
robots.txt
instead. Add it if you like — it's a cheap static file — but don't expect it to do the work.Boring, durable engineering:
curl
, not DevTools.robots.txt
; block only what you must.None of this is a hack. It's the same discipline as good, crawlable web engineering — which is why developers, not marketers, tend to be the ones who move AI visibility. It's the technical backbone of modern AI SEO and it overlaps almost entirely with classic technical SEO.
Build for the crawler that can't run your JavaScript, and you're ready for whatever the answer engines do next.
Sources: Cloudflare Radar, Vercel/MERJ crawler study, Ahrefs brand-mentions analysis, and OpenAI/Anthropic/Google bot docs (2025-2026).