# The 2026 AI-Crawler Reality Check: What the Data Says for Developers > Source: > Published: 2026-07-24 09:18:57+00:00 Everyone's writing think-pieces about "optimizing for AI search." I wanted numbers. So I pulled together the 2025-2026 data from Cloudflare, Vercel/MERJ, Ahrefs and the AI providers' own docs. Some of it genuinely changed how I build sites. Here's the reality check, dev to dev. This is the big one. A Vercel/MERJ analysis of **500M+ crawler fetches** found that **none** of the major dedicated AI crawlers execute JavaScript — not OpenAI's `GPTBot` /`OAI-SearchBot` /`ChatGPT-User` , not Anthropic's `ClaudeBot` , not `PerplexityBot` , not Meta or ByteDance's `Bytespider` . They fetch your raw HTML and leave. They *do* download JS files (~11.5% of ChatGPT's fetches, ~23.8% of Claude's) — they just never run them. The exception is **Google**: its Web Rendering Service is shared across Search and Gemini, so Google-Extended renders JS the same way Googlebot does. Translation: if your content only exists after client-side hydration, every AI engine except Google sees an empty shell. ``` # see what the bots see curl -A "GPTBot" https://yoursite.com/page | grep "your headline" ``` If that's empty, you need SSR, SSG, or prerendering. Cloudflare's data shows AI crawler activity up sharply year over year, with **GPTBot's volume up ~305%** and reaching **~11.7% of all crawler requests** by mid-2025. But roughly **86% of that crawling is for model training**, not live search retrieval. And the referrals? Lopsided doesn't cover it. Measured crawl-to-referral ratios hit **~38,065:1 for Anthropic** and **~1,091:1 for OpenAI**. They read enormously, cite rarely. AI tools still send **under 1%** of outbound web traffic today. By early 2026, roughly **68% of Google searches ended without a click**, and pages with an AI Overview above them saw click-through drop by around **60%**. Being "on page one" is worth less every quarter; being *the cited source* is worth more. `robots.txt` instead. Add it if you like — it's a cheap static file — but don't expect it to do the work.Boring, durable engineering: `curl` , not DevTools.`robots.txt` ; block only what you must.None of this is a hack. It's the same discipline as good, crawlable web engineering — which is why developers, not marketers, tend to be the ones who move AI visibility. It's the technical backbone of modern [AI SEO](https://www.com8.io/services/seo-agency-dubai/ai-seo) and it overlaps almost entirely with classic [technical SEO](https://www.com8.io/services/seo-agency-dubai/technical-seo). Build for the crawler that can't run your JavaScript, and you're ready for whatever the answer engines do next. *Sources: Cloudflare Radar, Vercel/MERJ crawler study, Ahrefs brand-mentions analysis, and OpenAI/Anthropic/Google bot docs (2025-2026).*