Classifying AI Crawlers and AI Chat User Fetches from Server Logs

Canonry, a self-hosted analytics tool, has built a traffic classifier that separates AI crawlers from user-triggered fetches in server logs, addressing the lack of a reliable self-hosted solution. During a seven-day window on canonry.ai, 947 requests used AI user-fetch identities, while 1,667 used four crawler identities; of 927 ChatGPT-User requests, only 64 matched OpenAI's May 17 IP snapshot, and 789 unmatched requests arrived in five hourly bursts probing paths like /.env and /.ssh/id_rsa, resembling a scanner. The tool, available in the Canonry repository, aims to keep crawls, user fetches, citations, and referrals from being conflated into one misleading number.

Research Classifying AI Crawlers and User Fetches from Server Logs Server logs are where AI crawlers and user fetches show up. I couldn't find a good self-hosted way to separate the two, so I built it into Canonry. When the fetch happens on the provider's network A question in ChatGPT or Claude goes to the provider, not your site. If the app needs a live page, it may send a separate request to your server. php user action - provider backend - your site When that request carries the expected user-agent and matches its published IP range, the path behaves like a server-side proxy. Your origin sees a provider-controlled address instead of the person's browser. The provider gets the page back and may use it in the answer. The person may never visit your site. Traditional browser analytics start after a page loads and executes JavaScript. In this server-side fetch path, the provider retrieves the HTML without running the page's JavaScript, so the GA4 browser tag and marketing pixels do not fire. The request appears in edge or origin logs while the analytics dashboard records no visit. OpenAI says https://developers.openai.com/api/docs/bots certain user actions may visit a page with ChatGPT-User . Anthropic says https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler Claude may use Claude-User to retrieve content for a user's query. These are different from automatic crawlers such as GPTBot , OAI-SearchBot , and ClaudeBot . OpenAI publishes the IP ranges used by ChatGPT-User https://openai.com/chatgpt-user.json , and Anthropic publishes its bot ranges https://claude.com/crawling/bots.json . The providers do not promise that every user-triggered fetch follows this server-side path. A user-agent by itself is only a claim, so unmatched requests have to remain unverified. One AI traffic number hides different events | Event | What happened | |---|---| | Crawler | A request claimed an automatic crawler identity | | User fetch | A request claimed an identity tied to a user action | | Citation | An answer linked to the site | | Referral | A browser visit was attributed to an AI source | Canonry reads server traffic from Cloudflare, Cloud Run, Vercel, and WordPress. It stores hourly rollups by identity, normalized path, response status, and verification state. User fetches stay separate from crawler traffic. Citations come from saved answer evidence. Referrals come from browser visits attributed to an AI source. What showed up on canonry.ai During one seven-day window, 947 requests used AI user-fetch identities on canonry.ai . Another 1,667 used four crawler identities. Of the 927 requests labeled ChatGPT-User , Canonry's stored May 17 IP snapshot matched 64. At 9:27 p.m. ET on August 13, I rechecked the same logs against OpenAI's current file and got 76 matches. The other 851 did not match that snapshot. Of those unmatched requests, 789 arrived in five hourly bursts. Nearly all returned a redirect or a 404 while probing paths such as /.env and /.ssh/id rsa . That pattern resembles a scanner using an AI user-agent, though the IP mismatch alone does not identify the caller. These are requests, not users or citations. The main job is keeping crawls, user fetches, citations, and referrals from becoming one misleading number. The traffic classifier and source integrations are in the Canonry repository https://github.com/Canonry/canonry . FAQ Does every ChatGPT or Claude question fetch a website? No. Every question reaches the provider's backend, but only some questions cause an external page fetch. Those requests can use ChatGPT-User or Claude-User. Why might the site see an AI provider's IP address? When the provider fetches a page server-side, the site's server sees its egress IP and user-agent instead of the person's browser. A match against the published range supports that identity. Does a user fetch prove that the page was cited? No. A fetch proves that a request reached the site. Citation tracking requires inspecting the answer and its source links. Continue with the platform. Inspect the technical workflow, run it on your own site, or add live visibility reporting to an agency portal.