How to check which AI crawlers actually hit a new WordPress site in week one

A developer published a guide for self-hosted WordPress operators to detect AI crawler traffic in a site's first week, recommending a custom nginx or Apache log format that captures ISO-8601 timestamps, full user agents, client IPs, status codes, bytes sent, request URIs and request times. The guide includes an awk script that tallies hits from GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and other named crawlers, and stresses that user agents are client-chosen strings, so forward-confirmed reverse DNS is needed to verify a crawler's identity.

A site goes live on Monday. On Friday the question arrives: have the AI crawlers found it yet? If the access log format was picked years ago to debug PHP, there is no answer in the file, and the first week is already gone. Two decisions have to be made before the first request: capture enough per request, and be able to tie a user agent to an IP you can check. Everything below assumes a self-hosted WordPress site where you control the web server. nginx, in a file included from the http block: /etc/nginx/conf.d/aiwatch.conf log format aiwatch '$time iso8601 $remote addr "$http user agent" ' '$request method $request uri $status $body bytes sent ' '"$http referer" $request time'; access log /var/log/nginx/aiwatch.log aiwatch; Apache httpd: LogFormat "%{%Y-%m-%dT%H:%M:%S%z}t %a \"%{User-Agent}i\" %m %U %s %B \"%{Referer}i\" %D" aiwatch CustomLog /var/log/apache2/aiwatch.log aiwatch Put it at the server level, not inside one vhost, or the next site you add silently logs in the old format. Then check what logrotate does to that file: if it rotates weekly and compresses, copy week one somewhere before it is replaced. | Field | What it decides | |---|---| | time iso8601 | Contains the UTC offset. Without it, cross-checking hits against the moment you published or changed robots.txt is guesswork. | | remote addr | Must be the client IP. Behind a CDN or proxy it is the edge, and your verification step will confirm the CDN as a crawler. Fix with set real ip from plus real ip header for whatever header your edge sets. | | http user agent | Store it whole. Truncating to 60 characters drops the version and the +https://... info URL that operators put there on purpose. | | status | 200, 304, 403 and 429 mean four different things. A crawler getting 403 is a different problem from a crawler that never came. | | body bytes sent | A 200 with 0 bytes is a HEAD request or an empty body. A status-only report calls both a success. | | request uri | Did it fetch /robots.txt first, and did it ask for /llms.txt at all. | | request time | A crawler that times out on your slowest page may not come back for the rest of the site. | The UA is field 2 when you split the line on double quotes, because the format wraps it in quotes. LOG=/var/log/nginx/aiwatch.log awk -F'"' ' { ua = tolower $2 if ua ~ /gptbot/ c = "GPTBot" else if ua ~ /oai-searchbot/ c = "OAI-SearchBot" else if ua ~ /chatgpt-user/ c = "ChatGPT-User" else if ua ~ /claudebot/ c = "ClaudeBot" else if ua ~ /claude-user/ c = "Claude-User" else if ua ~ /perplexitybot/ c = "PerplexityBot" else if ua ~ /perplexity-user/ c = "Perplexity-User" else if ua ~ /ccbot/ c = "CCBot" else if ua ~ /bytespider/ c = "Bytespider" else if ua ~ /amazonbot/ c = "Amazonbot" else if ua ~ /applebot/ c = "Applebot" else if ua ~ /googlebot/ c = "Googlebot" else next n c ++ } END { for k in n printf "%-16s %d\n", k, n k } ' "$LOG" | sort -k2 -nr A zero column for a crawler is not yet evidence of anything. Read the days before you read the total. To see whether a crawler walked the site in order, print the first hit per IP with the path it asked for: grep -i 'gptbot' "$LOG" | awk -F'"' '{split $1,a," " ; print a 1 , a 2 , $3}' | sort | head -20 With this format, $1 is timestamp + space + ip , and $3 is the request line method, path, status, bytes . If robots.txt and the home page appear in that list before anything else, the crawl is behaving; if the first hit is a deep product URL, it came in through a link, not through your sitemap. A user agent is a string the client chooses. Anyone can send GPTBot . Forward-confirmed reverse DNS is the cheap check: resolve the IP to a hostname, resolve that hostname back, and require the original IP to appear in the result. bash /usr/bin/env python3 """Forward-confirmed reverse DNS. Reads IPs from argv or stdin.""" import socket import sys def check ip : try: host = socket.gethostbyaddr ip 0 except socket.herror, socket.gaierror, OSError : return ip, None, False try: addrs = socket.gethostbyname ex host 2 except socket.gaierror, OSError : return ip, host, False return ip, host, ip in addrs def main : ips = sys.argv 1: or line.strip for line in sys.stdin if line.strip for ip in ips: addr, host, ok = check ip print f"{addr}\t{host or '-'}\t{'confirmed' if ok else 'NOT confirmed'}" if name == " main ": main chmod +x fcdns.py awk -F'"' 'tolower $2 ~ /gptbot|claudebot|perplexitybot/ {split $1,a," " ; print a 2 }' "$LOG" \ | sort -u | ./fcdns.py Confirmed PTR means the IP belongs to whoever controls that reverse zone. It does not prove the request came from the crawler you have in mind, so compare against the operator's published IP ranges where they publish one. A hit that fails both checks is an unknown client with a borrowed user agent, and it should not go into your report as a crawler visit. Google-Extended and Applebot-Extended are robots.txt user-agent tokens, not strings any client sends. Requests from those operators show up as Googlebot and Applebot . So "did Google-Extended fetch my page" has no answer in the access log; the only thing you control is whether robots.txt permits that token, which is a separate check against the file itself. Which brings up the other half of the question. A user agent in the log does not mean your robots.txt was obeyed. Pull the file you served that week and see what it said: curl -sS -o /dev/null -w '%{http code} %{content type}\n' https://example.com/robots.txt curl -sS https://example.com/llms.txt | head -20 Run it from outside your own network. A WAF rule that allows your office IP and blocks everything else looks identical to "no crawlers came" once you are reading the log. A fetch is not an inclusion. A confirmed GPTBot request for the home page says a crawler read the page; it says nothing about whether the page will appear in an answer later, and there is no field in the log that closes that gap. The reverse is worth stating too: no hits in week one does not distinguish "blocked" from "has not found it yet". A brand-new domain with no inbound links gives a crawler very little to follow, so an empty column is the expected starting state, not a malfunction. Do not rewrite robots.txt on the strength of one quiet week; check what the file actually serves first, then wait. Print the per-crawler count once a week and keep the output next to the raw log. A single snapshot after launch tells you nothing that changes a decision. The number that matters is the change between week one and week four, and whether the paths being fetched moved from the home page into the pages you actually want quoted. If maintaining this pipeline on every site is not the job you want, AumCrawl https://aumcreate.com/plugins/aumcrawl?utm source=devto&utm medium=content&utm campaign=devto-first-week-ai-crawler-check records crawler hits per crawler and lets you allow or block them individually. It is free and on wordpress.org; the log format above is still worth setting, because the plugin's data is only as good as what the server keeps.