Which Bots Show Up in Your Analytics? I Sent 10 Automated Visitors to a Test Page A developer ran a controlled experiment on 2026-11-20, sending ten kinds of automated clients to a one-page test site three times each and comparing server logs against JavaScript analytics beacons. HTML-only fetchers spoofing GPTBot, OAI-SearchBot, ChatGPT-User and PerplexityBot user agents produced 15 server-log entries and zero beacons, while headless Chrome and Playwright clients fired beacons and were detectable via the HeadlessChrome user agent and navigator.webdriver. A real Chrome window steered over the DevTools protocol without the automation flag matched a human visitor on user agent, client hints and navigator.webdriver, and its 20 synthetic mouse moves registered as ordinary pointer events. "How much AI traffic do we get?" sounds like one question, but it's really two. Some automated visitors only download your HTML: search and training crawlers, and the fetchers an assistant uses when a user asks about your page. Others drive a real browser: test scripts, scrapers, and the newer agents that click around a site on someone's behalf. The first kind shows up in one place, the second kind in another, and some of them look exactly like people. I wanted to see this directly instead of guessing. So on 2026-11-20 I set up a one-page test site on my own machine, sent ten kinds of automated client to it three times each, and compared two records: the server log every HTTP request and the analytics beacon a small JavaScript snippet that reports a page view, the way a JS analytics tracker does . The test page was plain HTML with an inline script. On load, the script sent a beacon with navigator.sendBeacon MDN https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon containing the user agent, navigator.webdriver , the client-hint brands, screen size and focus state. It counted pointer, scroll, key and touch events, and sent a second beacon when the page was left. A 34-line Python server logged every request and every beacon. The clients, all on Linux with Chrome 156: urllib . That shows how your stack treats those strings. It doesn't prove how the real crawlers behave. --headless=new --dump-dom , no automation library. --enable-automation flag. J also sent 20 synthetic mouse moves and a scroll. Each client visited three times. The scripts and raw logs are small, and nothing here depends on a particular analytics product. | Client | Server log HTML requests | Analytics beacons | User agent says | navigator.webdriver | |---|---|---|---|---| | A: GPTBot UA curl | 3 | 0 | GPTBot | n/a no JS | | B: OAI-SearchBot UA curl | 3 | 0 | Chrome 131 on macOS + "OAI-SearchBot" | n/a | | C: ChatGPT-User UA urllib | 3 | 0 | ChatGPT-User | n/a | | D: PerplexityBot UA curl | 3 | 0 | PerplexityBot | n/a | | E: curl | 3 | 0 | curl/8.14.1 | n/a | | F: Chrome --headless | 3 | 3 | HeadlessChrome/156 | false | | G: Playwright, headless | 3 | 3 +3 leave | HeadlessChrome/156 | true | | H: Playwright, window | 3 | 3 +3 leave | Chrome/156 | true | | I: Chrome over DevTools | 3 | 3 +3 leave | Chrome/156 | false | | J: same + mouse moves | 3 | 3 +3 leave | Chrome/156 | false | Three things stand out. 1. HTML-only fetchers never reach a JS tracker. Fifteen requests from A–E, fifteen lines in the server log, zero beacons. No script ran, so the analytics snippet never ran. If you've ever wondered why your analytics shows no GPTBot at all while your hosting bill shows plenty of bot traffic, this is the reason. Crawlers that do run JavaScript exist some search engines render pages in a browser engine , so don't read this as "bots never appear". But a client that only downloads HTML is invisible to a JS tracker by design. 2. Headless Chrome is easy to spot, unless you only read client hints. F and G put HeadlessChrome/156 in the user agent. But the Sec-CH-UA header and navigator.userAgentData.brands said "Google Chrome";v="156" for all five browser clients, headless or not. A filter that relies on client hints alone would count headless Chrome as a normal Chrome visit. 3. A real Chrome window steered by a program looked like a person. I and J sent the same user agent, the same client hints and navigator.webdriver === false , just like a human in Chrome. navigator.webdriver is the flag that browsers set when they are under automation MDN https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver . Playwright set it G, H . Steering Chrome over the DevTools protocol without the automation switch did not. And J's 20 synthetic mouse moves arrived as 20 ordinary pointer events, so an "engaged visitor" filter based on mouse movement would have counted it as engaged. Two details I'd treat as noise from my setup rather than signals: the headed clients reported one pointer event even without any input probably the cursor sitting over the new window , and I and J reported document.hasFocus === false , likely because my virtual display had no window manager. The time on page told me more: the scripted visits left after 1.50–1.52 seconds every time, and J after 2.85 seconds. Real people aren't that consistent. Most analytics and log tools sort traffic with a user-agent parser. I ran the documented strings through two popular Python libraries, ua-parser 1.0.2 and user-agents 2.2.0: | String | ua-parser family / OS / device | user-agents is bot | |---|---|---| | GPTBot | GPTBot / Other / Spider | True | | OAI-SearchBot | OAI-SearchBot / Mac OS X / Mac | False | | ChatGPT-User | "com/bot" / Other / Other | False | | PerplexityBot | PerplexityBot / Other / Spider | True | | HeadlessChrome | HeadlessChrome / Linux / Other | False | OAI-SearchBot's documented string starts like an ordinary Chrome-on-Mac user agent and adds compatible; OAI-SearchBot/1.4 at the end. In these versions, one library called it a Mac and the other didn't flag it as a bot. ChatGPT-User, which OpenAI describes as a user-triggered fetcher rather than a crawler, wasn't flagged either. Rule lists get updated, so your version may behave differently. That's exactly why it's worth testing the strings you care about. Count crawlers and fetchers from the server side. Your access log or your CDN's log is the only place HTML-only fetchers appear. Match the bot tokens before you match browser names, because some bot strings contain "Chrome". A minimal classifier: // Order matters: bot tokens first, because some bot UAs also contain "Chrome". const AI TOKENS = "GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User", "Claude-SearchBot", "PerplexityBot", "Perplexity-User" ; function classify ua = "" { const ai = AI TOKENS.find t = ua.includes t ; if ai return { kind: "ai-bot", name: ai }; if /bot|crawler|spider|curl|wget|python-requests|HeadlessChrome/i.test ua return { kind: "other-bot" }; return { kind: "browser" }; } Keep the token list in one place and check it against the vendors' pages now and then. Anthropic, for example, lists ClaudeBot, Claude-User and Claude-SearchBot and what each one is for in its help article https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler . A user agent is only a claim, though. OpenAI and Perplexity also publish IP ranges for their bots, and our earlier piece on counting AI agents in your access logs https://medium.com/@findutils/do-ai-agents-actually-read-your-llms-txt-count-it-in-your-access-logs-8224bc5d891f walks through checking a request against them. Tag automation in the tracker, but don't promise yourself it catches everything. In the browser, two cheap checks would have caught G and H and F's user agent : js const automated = navigator.webdriver === true || /HeadlessChrome/.test navigator.userAgent ; // send it along with the page view, e.g. { ..., automated } They would not have caught I or J. Nothing in the user agent, the client hints or the webdriver flag set them apart. For that kind of traffic, look at patterns instead of single visits: identical visit lengths, a burst of visits from one network, many pages in a few seconds, entry pages nobody links to. Annotate the day you add or change a filter, so that a sudden drop in "visitors" isn't mistaken for a real one. Segment, don't silently delete. If you throw away everything that looks automated, you can't check the filter later. Keep a "suspected automation" segment, or at least a count of what was excluded, and compare it with your human numbers when something looks odd. Check what your bots are allowed to fetch. Classifying bots is half the job. The other half is deciding which ones may crawl what. The free FindUtils Bot Crawl Checker https://findutils.com/network/bot-crawl-checker/ tells you whether your robots.txt allows or blocks a URL for 24 well-known search, AI and social bots and shows the rule that decides it. The robots.txt Tester https://findutils.com/seo/robots-txt-tester/ lets you try paths against a draft robots.txt, in your browser, before you deploy it. None of this needs cookies or personal data. User agent, a webdriver flag and a page path are enough to separate most automation, and the aggregate patterns above work on counts. That's also how DataVisitors https://datavisitors.com/ , the cookieless analytics product from the team behind FindUtils, describes its approach. Its web analytics page https://datavisitors.com/features/web-analytics says bot and referrer-spam filtering "removes known unwanted activity before storage", that refusal counts show what the filters exclude, and, honestly, that filtering "cannot guarantee that every unwanted request is detected". The same page describes an AI assistants channel for visits that arrive from ChatGPT, Claude, Perplexity, Gemini and Copilot with usable referrer information, which is the human side of AI traffic: people clicking through from an answer, not crawlers. Accounts are invitation-only for now, so the way in is the waitlist on the homepage. navigator.webdriver === true and HeadlessChrome ; don't rely on client hints alone. Need help with a project? The team behind FindUtils takes on freelance work on Upwork https://www.upwork.com/freelancers/~010a5875eab37b4bd7 , including PHP bug fixes and server-side bot filtering for a PHP site.