{"slug": "which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page", "title": "Which Bots Show Up in Your Analytics? I Sent 10 Automated Visitors to a Test Page", "summary": "A developer ran a controlled experiment on 2026-11-20, sending ten kinds of automated clients to a one-page test site three times each and comparing server logs against JavaScript analytics beacons. HTML-only fetchers spoofing GPTBot, OAI-SearchBot, ChatGPT-User and PerplexityBot user agents produced 15 server-log entries and zero beacons, while headless Chrome and Playwright clients fired beacons and were detectable via the HeadlessChrome user agent and navigator.webdriver. A real Chrome window steered over the DevTools protocol without the automation flag matched a human visitor on user agent, client hints and navigator.webdriver, and its 20 synthetic mouse moves registered as ordinary pointer events.", "body_md": "\"How much AI traffic do we get?\" sounds like one question, but it's really two. Some automated visitors only download your HTML: search and training crawlers, and the fetchers an assistant uses when a user asks about your page. Others drive a real browser: test scripts, scrapers, and the newer agents that click around a site on someone's behalf. The first kind shows up in one place, the second kind in another, and some of them look exactly like people.\n\nI wanted to see this directly instead of guessing. So on 2026-11-20 I set up a one-page test site on my own machine, sent ten kinds of automated client to it three times each, and compared two records: the **server log** (every HTTP request) and the **analytics beacon** (a small JavaScript snippet that reports a page view, the way a JS analytics tracker does).\n\nThe test page was plain HTML with an inline script. On load, the script sent a beacon with `navigator.sendBeacon()` ([MDN](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon)) containing the user agent, `navigator.webdriver`, the client-hint brands, screen size and focus state. It counted pointer, scroll, key and touch events, and sent a second beacon when the page was left. A 34-line Python server logged every request and every beacon.\n\nThe clients, all on Linux with Chrome 156:\n\n`urllib`. That shows how your stack treats those strings. It doesn't prove how the real crawlers behave.`--headless=new --dump-dom`), no automation library.`--enable-automation` flag. J also sent 20 synthetic mouse moves and a scroll.\nEach client visited three times. The scripts and raw logs are small, and nothing here depends on a particular analytics product.\n\n| Client | Server log (HTML requests) | Analytics beacons | User agent says | `navigator.webdriver` | \n|---|---|---|---|---|\n| A: GPTBot UA (curl) | 3 | 0 | GPTBot | n/a (no JS) | \n| B: OAI-SearchBot UA (curl) | 3 | 0 | Chrome 131 on macOS + \"OAI-SearchBot\" | n/a | \n| C: ChatGPT-User UA (urllib) | 3 | 0 | ChatGPT-User | n/a | \n| D: PerplexityBot UA (curl) | 3 | 0 | PerplexityBot | n/a | \n| E: curl | 3 | 0 | curl/8.14.1 | n/a | \n| F: Chrome --headless | 3 | 3 | HeadlessChrome/156 | false | \n| G: Playwright, headless | 3 | 3 (+3 leave) | HeadlessChrome/156 | **true** | \n| H: Playwright, window | 3 | 3 (+3 leave) | Chrome/156 | **true** | \n| I: Chrome over DevTools | 3 | 3 (+3 leave) | Chrome/156 | false | \n| J: same + mouse moves | 3 | 3 (+3 leave) | Chrome/156 | false | \n\nThree things stand out.\n\n**1. HTML-only fetchers never reach a JS tracker.** Fifteen requests from Aâ€“E, fifteen lines in the server log, zero beacons. No script ran, so the analytics snippet never ran. If you've ever wondered why your analytics shows no GPTBot at all while your hosting bill shows plenty of bot traffic, this is the reason. Crawlers that do run JavaScript exist (some search engines render pages in a browser engine), so don't read this as \"bots never appear\". But a client that only downloads HTML is invisible to a JS tracker by design.\n\n**2. Headless Chrome is easy to spot, unless you only read client hints.** F and G put `HeadlessChrome/156` in the user agent. But the `Sec-CH-UA` header and `navigator.userAgentData.brands` said `\"Google Chrome\";v=\"156\"` for all five browser clients, headless or not. A filter that relies on client hints alone would count headless Chrome as a normal Chrome visit.\n\n**3. A real Chrome window steered by a program looked like a person.** I and J sent the same user agent, the same client hints and `navigator.webdriver === false`, just like a human in Chrome. `navigator.webdriver` is the flag that browsers set when they are under automation ([MDN](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver)). Playwright set it (G, H). Steering Chrome over the DevTools protocol without the automation switch did not. And J's 20 synthetic mouse moves arrived as 20 ordinary pointer events, so an \"engaged visitor\" filter based on mouse movement would have counted it as engaged.\n\nTwo details I'd treat as noise from my setup rather than signals: the headed clients reported one pointer event even without any input (probably the cursor sitting over the new window), and I and J reported `document.hasFocus() === false`, likely because my virtual display had no window manager. The time on page told me more: the scripted visits left after 1.50â€“1.52 seconds every time, and J after 2.85 seconds. Real people aren't that consistent.\n\nMost analytics and log tools sort traffic with a user-agent parser. I ran the documented strings through two popular Python libraries, `ua-parser` 1.0.2 and `user-agents` 2.2.0:\n\n| String | ua-parser (family / OS / device) | user-agents `is_bot` | \n|---|---|---|\n| GPTBot | GPTBot / Other / Spider | True | \n| OAI-SearchBot | OAI-SearchBot / Mac OS X / Mac | **False** | \n| ChatGPT-User | \"com/bot\" / Other / Other | **False** | \n| PerplexityBot | PerplexityBot / Other / Spider | True | \n| HeadlessChrome | HeadlessChrome / Linux / Other | **False** | \n\nOAI-SearchBot's documented string starts like an ordinary Chrome-on-Mac user agent and adds `compatible; OAI-SearchBot/1.4` at the end. In these versions, one library called it a Mac and the other didn't flag it as a bot. ChatGPT-User, which OpenAI describes as a user-triggered fetcher rather than a crawler, wasn't flagged either. Rule lists get updated, so your version may behave differently. That's exactly why it's worth testing the strings you care about.\n\n**Count crawlers and fetchers from the server side.** Your access log (or your CDN's log) is the only place HTML-only fetchers appear. Match the bot tokens before you match browser names, because some bot strings contain \"Chrome\". A minimal classifier:\n\n```\n// Order matters: bot tokens first, because some bot UAs also contain \"Chrome\".\nconst AI_TOKENS = [\"GPTBot\", \"OAI-SearchBot\", \"ChatGPT-User\", \"ClaudeBot\", \"Claude-User\",\n  \"Claude-SearchBot\", \"PerplexityBot\", \"Perplexity-User\"];\nfunction classify(ua = \"\") {\n  const ai = AI_TOKENS.find((t) => ua.includes(t));\n  if (ai) return { kind: \"ai-bot\", name: ai };\n  if (/bot|crawler|spider|curl|wget|python-requests|HeadlessChrome/i.test(ua)) return { kind: \"other-bot\" };\n  return { kind: \"browser\" };\n}\n```\n\nKeep the token list in one place and check it against the vendors' pages now and then. Anthropic, for example, lists ClaudeBot, Claude-User and Claude-SearchBot and what each one is for in [its help article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). A user agent is only a claim, though. OpenAI and Perplexity also publish IP ranges for their bots, and our earlier piece on [counting AI agents in your access logs](https://medium.com/@findutils/do-ai-agents-actually-read-your-llms-txt-count-it-in-your-access-logs-8224bc5d891f) walks through checking a request against them.\n\n**Tag automation in the tracker, but don't promise yourself it catches everything.** In the browser, two cheap checks would have caught G and H (and F's user agent):\n\n``` js\nconst automated = navigator.webdriver === true || /HeadlessChrome/.test(navigator.userAgent);\n// send it along with the page view, e.g. { ..., automated }\n```\n\nThey would not have caught I or J. Nothing in the user agent, the client hints or the webdriver flag set them apart. For that kind of traffic, look at patterns instead of single visits: identical visit lengths, a burst of visits from one network, many pages in a few seconds, entry pages nobody links to. Annotate the day you add or change a filter, so that a sudden drop in \"visitors\" isn't mistaken for a real one.\n\n**Segment, don't silently delete.** If you throw away everything that looks automated, you can't check the filter later. Keep a \"suspected automation\" segment, or at least a count of what was excluded, and compare it with your human numbers when something looks odd.\n\n**Check what your bots are allowed to fetch.** Classifying bots is half the job. The other half is deciding which ones may crawl what. The free [FindUtils Bot Crawl Checker](https://findutils.com/network/bot-crawl-checker/) tells you whether your robots.txt allows or blocks a URL for 24 well-known search, AI and social bots and shows the rule that decides it. The [robots.txt Tester](https://findutils.com/seo/robots-txt-tester/) lets you try paths against a draft robots.txt, in your browser, before you deploy it.\n\nNone of this needs cookies or personal data. User agent, a webdriver flag and a page path are enough to separate most automation, and the aggregate patterns above work on counts.\n\nThat's also how [DataVisitors](https://datavisitors.com/), the cookieless analytics product from the team behind FindUtils, describes its approach. Its [web analytics page](https://datavisitors.com/features/web-analytics) says bot and referrer-spam filtering \"removes known unwanted activity before storage\", that refusal counts show what the filters exclude, and, honestly, that filtering \"cannot guarantee that every unwanted request is detected\". The same page describes an AI assistants channel for visits that arrive from ChatGPT, Claude, Perplexity, Gemini and Copilot with usable referrer information, which is the human side of AI traffic: people clicking through from an answer, not crawlers. Accounts are invitation-only for now, so the way in is the waitlist on the homepage.\n\n`navigator.webdriver === true` and `HeadlessChrome`; don't rely on client hints alone.\n*Need help with a project? The team behind FindUtils takes on freelance work on [Upwork](https://www.upwork.com/freelancers/~010a5875eab37b4bd7), including PHP bug fixes and server-side bot filtering for a PHP site.*", "url": "https://wpnews.pro/news/which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page", "canonical_source": "https://dev.to/findutils/which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page-3c1h", "published_at": "2026-11-20 12:38:05+00:00", "updated_at": "2026-11-20 12:42:26.309531+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-search", "artificial-intelligence"], "entities": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "PerplexityBot", "Playwright", "Chrome", "OpenAI", "Perplexity"], "also_reported_by": [], "takeaway": "HTML-only crawlers spoofing GPTBot, OAI-SearchBot, ChatGPT-User or PerplexityBot never fire a JS analytics beacon, and Chrome steered over DevTools without --enable-automation reports navigator.webdriver as false, so server logs plus automation flags are needed to catch them.", "recommendation": {"advice": "Log every HTTP request server-side rather than relying on JS beacons, and treat navigator.webdriver as unreliable for Chrome driven over DevTools.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page", "markdown": "https://wpnews.pro/news/which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page.md", "text": "https://wpnews.pro/news/which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page.txt", "jsonld": "https://wpnews.pro/news/which-bots-show-up-in-your-analytics-i-sent-10-automated-visitors-to-a-test-page.jsonld"}}