A Disallow under User-agent: * does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and * is only the fallback for crawlers you didn't name.
I learned this from my own logs. I had blocked my /go/ redirect links under *, and Cloudflare-verified Googlebot still fetched them 133 times in three days, ClaudeBot 213 times in five. Here's why, how to fix it, and a small script that catches the mistake before it ships.
User-agent: Googlebot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: *
Allow: /
Disallow: /go/
It's easy to end up here: you allow AI search crawlers by name, then later add a Disallow under *. In this file, /go/ is allowed for both Googlebot and ClaudeBot.
Both specs say so:
* group.
Groups don't inherit from each other, and * is not a shared base.
Write each blocked path into every named group.
User-agent: Googlebot
Allow: /
Disallow: /go/
User-agent: ClaudeBot
Allow: /
Disallow: /go/
User-agent: *
Allow: /
Disallow: /go/
Keeping Allow: / next to Disallow: /go/ is fine. RFC 9309, section 2.2.2, picks the longest matching rule, so for /go/amazon/... the four-character Disallow: /go/ beats the one-character Allow: /.
Once Googlebot re-read the fixed file, it stopped:
| Day (UTC) | Googlebot → /go/ |
|---|---|
| Oct 26 | 49 |
| Oct 27 (fixed at 13:31) | 52 |
| Oct 28 – Nov 2 | 0 |
It took about four hours to stop, which fits Google caching robots.txt for up to 24 hours.
With a dozen named groups, you will eventually miss one. This script collects every User-agent in the file and checks the paths you want blocked against each of them:
import re
import sys
from protego import Protego # pip install protego
def named_agents(text):
"""Every User-agent name in robots.txt except *."""
names = re.findall(r"(?im)^\s*user-agent\s*:\s*([^\s#]+)", text)
return sorted({n for n in names if n != "*"}, key=str.lower)
def check(text, base, paths):
rp = Protego.parse(text)
agents = named_agents(text) + ["SomeUnnamedBot"] # last one exercises the * group
leaks = []
for ua in agents:
for p in paths:
ok = rp.can_fetch(base + p, ua)
print(f"{ua:20s} {p:28s} {'allow' if ok else 'DISALLOW'}")
if ok:
leaks.append((ua, p))
return leaks
if __name__ == "__main__":
robots_file, base, *paths = sys.argv[1:]
leaks = check(open(robots_file, encoding="utf-8").read(), base, paths)
if leaks:
print(f"\n{len(leaks)} blocked path(s) are still allowed")
sys.exit(1)
python check_groups.py public/robots.txt https://example.com /go/amazon/B0CL6H35DY
On my old file, all 13 named groups came back allow and only SomeUnnamedBot was blocked, which is exactly what the logs showed. On the fixed file every row says DISALLOW and it exits 0. Put the paths you care about in CI and a new group without the Disallow will fail the build.
Python's standard urllib.robotparser says everything is allowed, even with the fixed file:
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.parse(open("robots.txt", encoding="utf-8").read().splitlines())
rp.can_fetch("Googlebot", "/go/amazon/B0CL6H35DY") # True, even after the fix
It reads a group top to bottom and stops at the first matching line. With Allow: / first, it never reaches Disallow: /go/. Use Protego, the longest-match parser Scrapy uses. I tested with 0.7.0.
My redirect links had rel="sponsored nofollow" and returned X-Robots-Tag: noindex, nofollow. Googlebot still fetched 67 different redirect URLs.
X-Robots-Tag comes back with the response, so by the time a crawler reads it, the request has already been made.
If you need the request itself to stop, use robots.txt.
The hour-by-hour data after the fix, why I can't credit the fix for ClaudeBot (it stopped two days earlier), and the unnamed GoogleOther that showed up anyway are on Aulvem → Aulvem | User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times