Distinguishing human readers from bot traffic with server-side stats

Independent blogger Tracy Durnell analyzed August 2026 AWStats server logs and found that "not viewed" bot traffic accounted for 320,678 pages, 379,023 hits and 11.36 GB of bandwidth against 878,149 human-viewed pages, 1,079,921 hits and 36.90 GB, concluding that AWStats alone cannot reliably separate human readers from AI scrapers. Durnell, who relies only on a robots.txt file rather than Cloudflare or Anubis blocking, estimated roughly 2,000 regular human readers by extrapolating from about 20 comments and webmentions at an assumed 1% response rate, plus 2,600 Hacker News referral pages and 4,000 referrals from search and other sources. Durnell said the exercise was motivated by the principle of AI scraping rather than bandwidth costs, since the hosting plan does not charge by bandwidth.

I was looking at my server stats AWStats to see if I could identify bot traffic easily. Spoilers: no. Basically, I wanted to know how “big” of an issue bot traffic is on my site since I’ve read some general reports from other bloggers. Some people are seeing less traffic https://clintmcmahon.com/blog/ai-killed-my-blog-traffic because they aren’t getting Google search referrals anymore https://sparktoro.com/blog/in-2026-less-than-one-third-of-google-searches-still-send-a-click/ ; some people are getting https://drewdevault.com/blog/Stop-externalizing-your-costs-on-me/ mobbed https://chrisglass.com/2025/11/13/combating-bots/ by bots https://www.coryd.dev/posts/2026/blocking-entire-countries-because-of-scrapers . I don’t use Cloudflare or Anubis to block suspicious traffic, just a robots.txt file where I futilely attempt to protect my site from the ravening horde of AI scrapers https://tracydurnell.com/2023/08/18/following-up-on-delisting-my-website-from-google/ . Using Vivaldi and DuckDuckGo browser, I am constantly deemed suspicious traffic and waiting on these verifications, and don’t want that for readers on my own site. My web hosting doesn’t charge based on bandwidth, so my interest is more about the principle of it https://tracydurnell.com/2023/07/11/pulling-my-site-from-google-over-ai-training/ . I haven’t found much guidance on deciphering AWStats, so I figured I’d write up my process of deduction. Admittedly, I am not a technical person and do not fully know what I am doing here 😅 Actually, I do know: I’m having fun I am a spreadsheet girl down to my core. I think looking at data and trying to tell stories from them can be a good way to learn though there’s potential danger in making bad assumptions . This has definitely showed me places that I don’t understand what the numbers represent, and how bots behave. Tell me if I’m drawing bad inferences anywhere 😉 Overall stats | August 2026 | | | | | | |---|---|---|---|---|---| | | Unique visitors | Number of visits | Pages | Hits | Bandwidth | |---|---|---|---|---|---| | Viewed traffic | 78,642 | 199,227 2.53 visits/visitor | 878,149 4.4 Pages/Visit | 1,079,921 5.42 Hits/Visit | 36.90 GB 194.22 KB/Visit | | Not viewed traffic | traffic generated by robots, worms, or replies with special HTTP status codes | | 320,678 | 379,023 | 11.36 GB | In actual terms, I consider these numbers imaginary. They are mostly useful for finding problems and seeing trends. Server stats are https://blog.jim-nielsen.com/2020/google-vs-netlify-analytics/ not comparable https://daverupert.com/2022/04/server-side-vs-client-side-analytics/ to numbers from client-side analytics like Google Analytics. They use different methods of measurement. I don’t have client-side analytics https://indieweb.org/analytics hooked up even though there are privacy-respecting alternatives to Google Analytics . I think playing with data is fun, but aside from troubleshooting, I don’t need to know how many people are reading my site. How many people read my site? To make a wild guess towards real numbers of human visitors: - Assuming ~1% https://en.wikipedia.org/wiki/1%25 rule of regular readers leave comments / webmentions or email me, I heard from about 20 people over the past month, which would be 2000 “regular” readers both on-site and via RSS . Being part of the indie web, people who read my site may be more inclined to comment than the typical ratio, though. - I had one old post that got a lot of visits from Hacker News; assuming that no one there regularly reads my website, the stats list 2600 referral pages. I don’t know if that count aggregates total views from anyone referred to the site through that link or the number of original clickthroughs — for this guesstimate I’m assuming the latter. - Between other sources and search, there are another 4000 referrals. I recognize most of the referral domains, so I think this is largely humans clicking links. The search traffic https://tracydurnell.com/2025/10/10/question-my-sites-not-indexed-but-im-getting-referrals-from-google/ I’m skeptical about but will include. So I’d guess 8000-10000 unique human visitors based on this quite light evidence, or about 10% of what the stats show. I’m using this as my benchmark number as I go through all the stats and think about what might be a filter for bots versus humans. What’s a bot, who’s human? What’s the best approach here: Identify the bots, or identify human traffic and assume the rest is bots? Traffic trends | August 2025 | | | | | | |---|---|---|---|---|---| | | Unique visitors | Number of visits | Pages | Hits | Bandwidth | |---|---|---|---|---|---| | Viewed traffic | 31,479 | 104,080 3.3 visits/visitor | 401,896 3.86 Pages/Visit | 579,158 5.56 Hits/Visit | 33.29 GB 335.35 KB/Visit | | Not viewed traffic | | | 368,751 | 404,197 | 17.56 GB | All these stats except bandwidth have roughly doubled compared to this month last year 2025 shown here, scroll back up for 2026 . Not sure that tells me anything without making wild extrapolations. The traffic that AWStats affirmatively classes as bots is actually higher last year 🤷♀️ I’m not sure what it means that visits and pages are up but bandwidth is only slightly higher much lower per visit — maybe I set up caching correctly since last year? Visit length | Visit duration | | | |---|---|---| | Average: 248 s | Number of visits Total = 199,227 | Percent | |---|---|---| | 0s-30s | 178,599 | 89.6 % | | 30s-2mn | 1,419 | 0.7 % | | 2mn-5mn | 1,110 | 0.5 % | | 5mn-15mn | 2,364 | 1.1 % | | 15mn-30mn | 2,776 | 1.3 % | | 30mn-1h | 5,930 | 2.9 % | | 1h+ | 7,029 | 3.5 % | 90% of “visits” to my site are less than 30 seconds, so I’m assuming those are basically all bots. Some are reader bots fetching for humans I suspect there are a lot of you out there in RSS land — but I don’t have a way to distinguish feed reader bots from other types of bots, so I’m lumping them together. Excluding the 0-30 second visits cuts it down to ~20,500 visits. If I knew whether bots tend to visit once a month or every day, I could guess whether 2.5 visits per unique visitor is low or high. Using 2.5 as the denominator would yield ~8200 unique visitors, in line with my wild guess. Browser | Browsers | Pages Total = 878,149 | Percent | |---|---|---| | Unknown | 581,915 | 66.2 % | | Google Chrome | 167,041 | 19 % | | Mozilla | 35,305 | 4 % | | NetNewsWire RSS Reader | 28,401 | 3.2 % | | Firefox | 27,679 | 3.1 % | | Safari | 19,956 | 2.2 % | | Android browser | 10,839 | 1.2 % | | Akregator RSS Reader | 1,879 | 0.2 % | | MS Internet Explorer | 1,645 | 0.1 % | | Netscape | 1,024 | 0.1 % | | Others | 2465 | 0.2 % | I feel like showing unique visitors per browser would be more useful info but maybe people use this to decide which browsers to support? We could remove NetNewsWire and Akregator as known reader bots, but that’s not much. I’m not sure whether “unknown” means it’s someone with good privacy settings or a bot. I don’t think I can draw any conclusions here , but I want to encourage whoever is using Netscape and IE to update to a modern browser 😉 File types | File type | Hits | |---|---| | HTML | 669,122 | | PHP | 32,658 | | CSS | 118,146 | | js | 37,472 | | ttf | 42,142 | I’m making an assumption here that each visitor only loads each unique file once , unless they refresh or clear their cache. Fonts feel like a possible avenue for differentiating readers from bots… I have 6 ttf fonts that a new visitor might load. If we assumed everyone loaded all the fonts not the case for every post and divided by 6, that would be 7000, not out of line with my guess. I don’t understand the behavior of bots well enough to know whether they load fonts or not. Under the separate page stats, there are listings for fonts… I had originally assumed that represented anyone who loaded the font, but I think that’s wrong. Is it scrapers that open every resource including the font file? Sounds like bots might not typically load javascript https://www.peterbe.com/plog/bot-traffic-hitting-my-blog , so could we use js as some sort of human indicator? Some people block javascript, so this would be a minimum estimation. I don’t know how javascript works with caching, does each hit on a new page or post trigger a hit on the js file again, assuming it’s on multiple pages? Outliers My own IP is assessed at visiting ~19,500 pages on my site in August, which averages out to 670 pages a day. Even considering how often I save a draft post, seems not possible. But disregarding the number accuracy , let’s assume that I visit my site more than any other human . There’s one IP address listed as viewing more pages than me… dramatically. Precisely 423,059 pages in August. This is clearly a bot of some variety. I feel we could safely exclude those pages from the total pages as obviously not human. With a total 878,149 pages for the month, that’s half knocked off right there Removing myself and MyBiggestFanBot, that cuts it down to ~435,000 pages. But I don’t think that number is reliable; what we could probably say comfortably is that at least half of the pages accessed are by bots in reality, more, because this is a single IP we’re talking about . Page:hit ratios I notice there are some IPs that list the exact same hits and pages, while others list dramatically more hits than views . I wonder if one of those ratios might be a screening factor? Or is that like, some people only looked at text-only posts, and other people looked at image-heavy posts? Entry and exit numbers I’m assuming that the much lower entry numbers are a closer approximation of human readers. Here’s a few actual pages and posts: Page entries and exits | Page name | Viewed | Entry | Exit | |---|---|---|---| | Seattle Area Activity Ideas https://tracydurnell.com/places/seattle-activity-ideas/ | 82,306 | 30 | 31 | | Seattle Travel Ideas https://tracydurnell.com/places/seattle-travel-ideas/ | 41,786 | 31 | 28 | | Blogroll https://tracydurnell.com/blogroll/ | 18,192 | 229 | 199 | | Wildlife sightings in my garden https://tracydurnell.com/garden/wildlife/ | 11,938 | 6 | 6 | The viewed numbers on the pages are obviously bullshit. It is absolutely ludicrous that 12,000 people looked at my wildlife page; 6 feels legit 😂 Post entries and exits | Post name | Viewed | Entry | Exit | |---|---|---|---| | Building community out of strangers https://tracydurnell.com/2023/11/30/building-community-out-of-strangers/ | 30,524 | 20,465 | 17,372 | | Authenticating digital works without “detectors” https://tracydurnell.com/2026/08/17/authenticating-digital-works/ | 1,097 | 442 | 256 | | What I read online today: Aug. 24, 2026 https://tracydurnell.com/2026/08/24/what-i-read-online-today-aug-24-2026/ | 806 | 195 | 171 | | Why I don’t do atomic notes https://tracydurnell.com/2026/08/13/why-i-dont-do-atomic-notes/ | 731 | 274 | 183 | | Weeknotes: Aug. 8-14, 2026 https://tracydurnell.com/2026/08/14/weeknotes-aug-8-14-2026/ | 621 | 180 | 142 | I could believe that 150-200 people open my weeknotes regularly, though I would be surprised if another 450 people found their way to my weeknotes after landing elsewhere on my site. I think any number that’s 1000ish or below is potentially plausible, but higher listings are being hit by bots. So potentially you might set a “minimum” understanding of human traffic based on normal-looking numbers and excluding obvious outliers. Referral traffic The article shared on Hacker News tallies 20,500 entries and 17,000 exits, while the domain only referred 2,600 visits 🤔 Even 2,600 people opening an article feels unlikely but given the source is possible. Between those numbers, I’d trust the referrals. The other referral numbers seem plausible — e.g. 170 pages from Kagi search, 130 from Bubbles, 50 from blogroll.org. So non-search referrals are the only visits I feel pretty confident in classing human . So, a substantial portion of my traffic is from bots some of which are feed reader bots fetching for human readers , but I couldn’t tell you if that means 50% or 90% — likely 50% is a minimum threshold but I would guess much higher. The results don’t really matter one way or the other — but it was a fun exercise in reasoning Related reading: running a personal website is not fun anymore https://winnielim.org/journal/running-a-personal-website-is-not-fun-anymore/ by Winnie Lim