{"slug": "user-agent-doesn-t-apply-to-crawlers-you-named-in-robots-txt", "title": "User-agent: * doesn't apply to crawlers you named in robots.txt", "summary": "A developer found that a Disallow rule under the wildcard User-agent: * group in robots.txt does not apply to crawlers named in their own groups, after logs showed Googlebot fetched blocked /go/ redirect links 133 times in three days and ClaudeBot 213 times in five. The fix is to repeat each blocked path inside every named group, since groups do not inherit from one another and RFC 9309 picks the longest matching rule. The developer also released a Python script built on the Protego parser that checks every named group for leaked paths and can fail a CI build, noting that Python's standard urllib.robotparser incorrectly reports everything as allowed.", "body_md": "A `Disallow` under `User-agent: *` does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and `*` is only the fallback for crawlers you didn't name.\n\nI learned this from my own logs. I had blocked my `/go/` redirect links under `*`, and Cloudflare-verified Googlebot still fetched them 133 times in three days, ClaudeBot 213 times in five. Here's why, how to fix it, and a small script that catches the mistake before it ships.\n\n```\nUser-agent: Googlebot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: *\nAllow: /\nDisallow: /go/\n```\n\nIt's easy to end up here: you allow AI search crawlers by name, then later add a Disallow under `*`. In this file, `/go/` is allowed for both Googlebot and ClaudeBot.\n\nBoth specs say so:\n\n`*` group.\nGroups don't inherit from each other, and `*` is not a shared base.\n\nWrite each blocked path into every named group.\n\n```\nUser-agent: Googlebot\nAllow: /\nDisallow: /go/\n\nUser-agent: ClaudeBot\nAllow: /\nDisallow: /go/\n\nUser-agent: *\nAllow: /\nDisallow: /go/\n```\n\nKeeping `Allow: /` next to `Disallow: /go/` is fine. RFC 9309, section 2.2.2, picks the longest matching rule, so for `/go/amazon/...` the four-character `Disallow: /go/` beats the one-character `Allow: /`.\n\nOnce Googlebot re-read the fixed file, it stopped:\n\n| Day (UTC) | Googlebot → `/go/` | \n|---|---|\n| Oct 26 | 49 | \n| Oct 27 (fixed at 13:31) | 52 | \n| Oct 28 – Nov 2 | 0 | \n\nIt took about four hours to stop, which fits Google caching robots.txt for up to 24 hours.\n\nWith a dozen named groups, you will eventually miss one. This script collects every `User-agent` in the file and checks the paths you want blocked against each of them:\n\n``` python\nimport re\nimport sys\nfrom protego import Protego  # pip install protego\n\ndef named_agents(text):\n    \"\"\"Every User-agent name in robots.txt except *.\"\"\"\n    names = re.findall(r\"(?im)^\\s*user-agent\\s*:\\s*([^\\s#]+)\", text)\n    return sorted({n for n in names if n != \"*\"}, key=str.lower)\n\ndef check(text, base, paths):\n    rp = Protego.parse(text)\n    agents = named_agents(text) + [\"SomeUnnamedBot\"]  # last one exercises the * group\n    leaks = []\n    for ua in agents:\n        for p in paths:\n            ok = rp.can_fetch(base + p, ua)\n            print(f\"{ua:20s} {p:28s} {'allow' if ok else 'DISALLOW'}\")\n            if ok:\n                leaks.append((ua, p))\n    return leaks\n\nif __name__ == \"__main__\":\n    robots_file, base, *paths = sys.argv[1:]\n    leaks = check(open(robots_file, encoding=\"utf-8\").read(), base, paths)\n    if leaks:\n        print(f\"\\n{len(leaks)} blocked path(s) are still allowed\")\n        sys.exit(1)\npython check_groups.py public/robots.txt https://example.com /go/amazon/B0CL6H35DY\n```\n\nOn my old file, all 13 named groups came back allow and only `SomeUnnamedBot` was blocked, which is exactly what the logs showed. On the fixed file every row says DISALLOW and it exits 0. Put the paths you care about in CI and a new group without the Disallow will fail the build.\n\nPython's standard `urllib.robotparser` says everything is allowed, even with the fixed file:\n\n``` python\nfrom urllib.robotparser import RobotFileParser\n\nrp = RobotFileParser()\nrp.parse(open(\"robots.txt\", encoding=\"utf-8\").read().splitlines())\nrp.can_fetch(\"Googlebot\", \"/go/amazon/B0CL6H35DY\")  # True, even after the fix\n```\n\nIt reads a group top to bottom and stops at the first matching line. With `Allow: /` first, it never reaches `Disallow: /go/`. Use [Protego](https://github.com/scrapy/protego), the longest-match parser Scrapy uses. I tested with 0.7.0.\n\nMy redirect links had `rel=\"sponsored nofollow\"` and returned `X-Robots-Tag: noindex, nofollow`. Googlebot still fetched 67 different redirect URLs.\n\n`X-Robots-Tag` comes back with the response, so by the time a crawler reads it, the request has already been made.\nIf you need the request itself to stop, use robots.txt.\n\nThe hour-by-hour data after the fix, why I can't credit the fix for ClaudeBot (it stopped two days earlier), and the unnamed GoogleOther that showed up anyway are on Aulvem → [Aulvem | User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times](https://aulvem.com/blog/2026-11-02-robots-txt-named-groups-ignore-wildcard/)", "url": "https://wpnews.pro/news/user-agent-doesn-t-apply-to-crawlers-you-named-in-robots-txt", "canonical_source": "https://dev.to/aulvem/user-agent-doesnt-apply-to-crawlers-you-named-in-robotstxt-12fi", "published_at": "2026-11-02 10:07:04+00:00", "updated_at": "2026-11-02 10:23:01.100617+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-search", "developer-tools"], "entities": ["Googlebot", "ClaudeBot", "Protego", "Scrapy", "Cloudflare", "RFC 9309"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/user-agent-doesn-t-apply-to-crawlers-you-named-in-robots-txt", "markdown": "https://wpnews.pro/news/user-agent-doesn-t-apply-to-crawlers-you-named-in-robots-txt.md", "text": "https://wpnews.pro/news/user-agent-doesn-t-apply-to-crawlers-you-named-in-robots-txt.txt", "jsonld": "https://wpnews.pro/news/user-agent-doesn-t-apply-to-crawlers-you-named-in-robots-txt.jsonld"}}