User-agent: * doesn't apply to crawlers you named in robots.txt

A developer found that a Disallow rule under the wildcard User-agent: * group in robots.txt does not apply to crawlers named in their own groups, after logs showed Googlebot fetched blocked /go/ redirect links 133 times in three days and ClaudeBot 213 times in five. The fix is to repeat each blocked path inside every named group, since groups do not inherit from one another and RFC 9309 picks the longest matching rule. The developer also released a Python script built on the Protego parser that checks every named group for leaked paths and can fail a CI build, noting that Python's standard urllib.robotparser incorrectly reports everything as allowed.

A Disallow under User-agent: does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and is only the fallback for crawlers you didn't name. I learned this from my own logs. I had blocked my /go/ redirect links under , and Cloudflare-verified Googlebot still fetched them 133 times in three days, ClaudeBot 213 times in five. Here's why, how to fix it, and a small script that catches the mistake before it ships. User-agent: Googlebot Allow: / User-agent: ClaudeBot Allow: / User-agent: Allow: / Disallow: /go/ It's easy to end up here: you allow AI search crawlers by name, then later add a Disallow under . In this file, /go/ is allowed for both Googlebot and ClaudeBot. Both specs say so: group. Groups don't inherit from each other, and is not a shared base. Write each blocked path into every named group. User-agent: Googlebot Allow: / Disallow: /go/ User-agent: ClaudeBot Allow: / Disallow: /go/ User-agent: Allow: / Disallow: /go/ Keeping Allow: / next to Disallow: /go/ is fine. RFC 9309, section 2.2.2, picks the longest matching rule, so for /go/amazon/... the four-character Disallow: /go/ beats the one-character Allow: / . Once Googlebot re-read the fixed file, it stopped: | Day UTC | Googlebot → /go/ | |---|---| | Oct 26 | 49 | | Oct 27 fixed at 13:31 | 52 | | Oct 28 – Nov 2 | 0 | It took about four hours to stop, which fits Google caching robots.txt for up to 24 hours. With a dozen named groups, you will eventually miss one. This script collects every User-agent in the file and checks the paths you want blocked against each of them: python import re import sys from protego import Protego pip install protego def named agents text : """Every User-agent name in robots.txt except .""" names = re.findall r" ?im ^\s user-agent\s :\s ^\s + ", text return sorted {n for n in names if n = " "}, key=str.lower def check text, base, paths : rp = Protego.parse text agents = named agents text + "SomeUnnamedBot" last one exercises the group leaks = for ua in agents: for p in paths: ok = rp.can fetch base + p, ua print f"{ua:20s} {p:28s} {'allow' if ok else 'DISALLOW'}" if ok: leaks.append ua, p return leaks if name == " main ": robots file, base, paths = sys.argv 1: leaks = check open robots file, encoding="utf-8" .read , base, paths if leaks: print f"\n{len leaks } blocked path s are still allowed" sys.exit 1 python check groups.py public/robots.txt https://example.com /go/amazon/B0CL6H35DY On my old file, all 13 named groups came back allow and only SomeUnnamedBot was blocked, which is exactly what the logs showed. On the fixed file every row says DISALLOW and it exits 0. Put the paths you care about in CI and a new group without the Disallow will fail the build. Python's standard urllib.robotparser says everything is allowed, even with the fixed file: python from urllib.robotparser import RobotFileParser rp = RobotFileParser rp.parse open "robots.txt", encoding="utf-8" .read .splitlines rp.can fetch "Googlebot", "/go/amazon/B0CL6H35DY" True, even after the fix It reads a group top to bottom and stops at the first matching line. With Allow: / first, it never reaches Disallow: /go/ . Use Protego https://github.com/scrapy/protego , the longest-match parser Scrapy uses. I tested with 0.7.0. My redirect links had rel="sponsored nofollow" and returned X-Robots-Tag: noindex, nofollow . Googlebot still fetched 67 different redirect URLs. X-Robots-Tag comes back with the response, so by the time a crawler reads it, the request has already been made. If you need the request itself to stop, use robots.txt. The hour-by-hour data after the fix, why I can't credit the fix for ClaudeBot it stopped two days earlier , and the unnamed GoogleOther that showed up anyway are on Aulvem → Aulvem | User-agent: doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times https://aulvem.com/blog/2026-11-02-robots-txt-named-groups-ignore-wildcard/