AI crawlers in WordPress logs: GPTBot shows up, Google-Extended never does

A developer has published a WordPress mu-plugin that logs visits from nine documented AI crawler user agents, after finding that Google-Extended and Applebot-Extended never appear in access logs because neither sends HTTP requests. The writeup maps OpenAI's GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, Anthropic's ClaudeBot, Claude-User and Claude-SearchBot, and Perplexity's PerplexityBot and Perplexity-User to their vendor-documented purposes and robots.txt behavior, noting that user-initiated fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt rules. The author discloses developing Prime SEO, a WordPress plugin that includes a similar log.

Google-Extended and Applebot-Extended, two of the names that robots.txt guides list most often for AI, cannot appear in a WordPress access log. Neither one sends a request. Google's crawler documentation https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers states that Google-Extended "doesn't have a separate HTTP request user agent string." The second name gets even less: "Applebot-Extended does not crawl webpages," Apple's support page for Applebot https://support.apple.com/en-us/119829 reads, which leaves both names as robots.txt tokens rather than visitors and means that a log of AI crawlers in WordPress has to be built from the agents that send requests. Disclosure: the author develops Prime SEO, a WordPress plugin that includes such a log. Every user agent below was checked against vendor documentation on September 21, 2026, and the code runs without any plugin. OpenAI lists four agents on its crawler page https://developers.openai.com/api/docs/bots . GPTBot gathers training material the page says it exists "to make our generative AI foundation models more useful and safe" , while OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page when a person asks for it. OAI-AdsBot checks pages submitted as ads. Anthropic documents three https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler , split the same way: ClaudeBot collects training data, while Claude-User and Claude-SearchBot serve live questions and search. Perplexity runs two. | User agent substring | Operator | Purpose, per the vendor | robots.txt | |---|---|---|---| | GPTBot | OpenAI | Model training | Honored | | OAI-SearchBot | OpenAI | ChatGPT search index | Honored | | ChatGPT-User | OpenAI | Fetch started by a user | "may not apply" | | OAI-AdsBot | OpenAI | Safety check of pages submitted as ads | Not stated | | ClaudeBot | Anthropic | Model training | Honored | | Claude-User | Anthropic | Fetch started by a user | Honored | | Claude-SearchBot | Anthropic | Search quality | Honored | | PerplexityBot | Perplexity | Perplexity search results | Honored | | Perplexity-User | Perplexity | Fetch started by a user | "generally ignores" | Missing from Anthropic's page are anthropic-ai and Claude-Web, two names that still circulate in robots.txt templates. A rule aimed at them blocks nothing the vendor admits to sending. Crawlers that build an index obey robots.txt, by the vendors' own account. Anthropic's help page says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Fetchers started by a person are a different matter. OpenAI describes ChatGPT-User as an agent that "is not used for crawling the web in an automatic fashion," and adds that "robots.txt rules may not apply" because a user starts the action. Perplexity's bot guide https://docs.perplexity.ai/guides/bots is blunter about Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." For a site owner, the split has a practical side. A Disallow line can keep a page out of a training set, yet the same page may still be fetched for a ChatGPT user five minutes later. Only a log shows that second kind of visit. WordPress needs no plugin for a basic log. The file below goes into wp-content/mu-plugins/ , hooks into template redirect at priority 1, and compares the user agent of each request with the nine documented names. <?php / Plugin Name: AI Crawler Log Description: Records visits from documented AI user agents. / add action 'template redirect', function { if empty $ SERVER 'HTTP USER AGENT' { return; } $agent = sanitize text field wp unslash $ SERVER 'HTTP USER AGENT' ; $names = array 'GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'OAI-AdsBot', 'ClaudeBot', 'Claude-User', 'Claude-SearchBot', 'PerplexityBot', 'Perplexity-User', ; foreach $names as $name { if false === stripos $agent, $name { continue; } $log = get option 'ai crawler log', array ; $log = array 'bot' = $name, 'url' = esc url raw wp unslash $ SERVER 'REQUEST URI' ?? '/' , 'time' = gmdate 'Y-m-d H:i:s' , ; // Keep the last 500 visits; "false" keeps the option out of autoload. update option 'ai crawler log', array slice $log, -500 , false ; break; } }, 1 ; Each match lands in a single option, ai crawler log , stored with autoload switched off so that ordinary page loads never carry it. The array slice call caps the record at 500 visits. On a local WordPress 7.1 test install, a request sent with the GPTBot agent string appeared in the option within the same second, while a Chrome agent left no trace. Reading the result takes one WP-CLI command: wp option get ai crawler log --format=json . Owners with shell access can skip PHP and count matches in the server log instead: grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn PHP-based logging has a blind spot. When a full-page cache WP Rocket or a CDN edge, for instance answers the request, WordPress never starts and the visit goes unrecorded. The server access log catches those hits; a PHP log on a cached site shows only cache misses. User agent strings are also trivial to fake. Any script can call itself GPTBot, which is why all three vendors publish their IP ranges as JSON files openai.com/gptbot.json is one of them . A visit from outside those ranges is an impostor. The opposite error exists too. Cloudflare reported on August 4, 2025 https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ that Perplexity used, besides its declared agents, "a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked." No list of names catches a crawler that refuses to give one. Traffic from these agents is no longer marginal. GPTBot "increased its share from 2.2% to 7.7%" of crawler requests in the year to May 2025, João Tomé of Cloudflare wrote in a study of crawler traffic https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ published that July. Raw request volume from the same bot, the figure in the heading above, more than quadrupled. Growth was uneven. In the same dataset ClaudeBot, Anthropic's training crawler, "fell from 11.7% to 5.4% of total traffic." One site's log can look nothing like the global chart, and a quiet month for one vendor says little about the others. Blocking remains rare among large sites. Of the 3,816 robots.txt files that Cloudflare could read among top domains, 546 addressed AI bots at all. GPTBot led the refusals: the study found it "disallowed by 312 domains." Prime SEO, the plugin named in the disclosure, ships the same stripos approach behind a dashboard screen called AI Crawler Stats, which also records requests for llms.txt and the XML sitemap. Version 1.1.9 matches 14 substrings. Two of them are Google-Extended and Applebot-Extended, which, per the documentation quoted above, can never match. Those two counters will stay at zero. The plugin's list also predates Claude-User, Claude-SearchBot and Perplexity-User, so it undercounts user-triggered visits today; an update is planned for version 1.2.0. Background on the file these agents are meant to read is in the guide to what llms.txt is https://prime-seo-plugin.com/blog/what-is-llms-txt/ and on the llms.txt plugin for WordPress https://prime-seo-plugin.com/llms-txt-wordpress-plugin/ page. All three vendors now separate training crawlers from agents that act for a person. If that split continues, a robots.txt file written for GPTBot and ClaudeBot alone will govern a shrinking part of AI traffic, and the raw access log not the rule file becomes the place where a new agent name first shows up. Until a vendor page changes, nine names are enough. Originally published on the Prime SEO blog https://prime-seo-plugin.com/blog/ai-crawlers-wordpress-logs/ .