…
Websites that disallow specific crawlers in robots.txt, from AI trainers like GPTBot and ClaudeBot to SEO and search bots like AhrefsBot and Bingbot. Based on the 880,858 websites tracked by Site Stats Database.
883 blocked crawlers & bots are used by 100 or more of the 125,901 sites we track in this dimension. Pick one to see the top websites, ranked by traffic.
| # | Name | Websites |
|---|---|---|
| 1 | ||
| 4 |
|
| 45,564 |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
| 9 | |
| 10 | |
| 11 | |
| 12 | |
| 13 | |
| 14 | |
| 15 | |
| 16 | |
| 17 | |
| 18 | |
| 19 | |
| 20 | |
| 21 | |
| 22 | |
| 23 | |
| 24 | |
| 25 | |
| 26 | |
| 27 | |
| 28 | |
| 29 | |
| 30 | |
| 31 | |
| 32 | |
| 33 | |
| 34 | |
| 35 | |
| 36 | |
| 37 | |
| 38 | |
| 39 | |
| 40 | |
| 41 | |
| 42 | |
| 43 | |
| 44 | |
| 45 | |
| 46 | |
| 47 | |
| 48 | |
| 49 | |
| 50 | |
source & further reading
sitestatsdb.com — original article