# User-agent: * doesn't apply to crawlers you named in robots.txt

> Source: <https://dev.to/aulvem/user-agent-doesnt-apply-to-crawlers-you-named-in-robotstxt-12fi>
> Published: 2026-11-02 10:07:04+00:00

A `Disallow` under `User-agent: *` does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and `*` is only the fallback for crawlers you didn't name.

I learned this from my own logs. I had blocked my `/go/` redirect links under `*`, and Cloudflare-verified Googlebot still fetched them 133 times in three days, ClaudeBot 213 times in five. Here's why, how to fix it, and a small script that catches the mistake before it ships.

```
User-agent: Googlebot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /
Disallow: /go/
```

It's easy to end up here: you allow AI search crawlers by name, then later add a Disallow under `*`. In this file, `/go/` is allowed for both Googlebot and ClaudeBot.

Both specs say so:

`*` group.
Groups don't inherit from each other, and `*` is not a shared base.

Write each blocked path into every named group.

```
User-agent: Googlebot
Allow: /
Disallow: /go/

User-agent: ClaudeBot
Allow: /
Disallow: /go/

User-agent: *
Allow: /
Disallow: /go/
```

Keeping `Allow: /` next to `Disallow: /go/` is fine. RFC 9309, section 2.2.2, picks the longest matching rule, so for `/go/amazon/...` the four-character `Disallow: /go/` beats the one-character `Allow: /`.

Once Googlebot re-read the fixed file, it stopped:

| Day (UTC) | Googlebot → `/go/` | 
|---|---|
| Oct 26 | 49 | 
| Oct 27 (fixed at 13:31) | 52 | 
| Oct 28 – Nov 2 | 0 | 

It took about four hours to stop, which fits Google caching robots.txt for up to 24 hours.

With a dozen named groups, you will eventually miss one. This script collects every `User-agent` in the file and checks the paths you want blocked against each of them:

``` python
import re
import sys
from protego import Protego  # pip install protego

def named_agents(text):
    """Every User-agent name in robots.txt except *."""
    names = re.findall(r"(?im)^\s*user-agent\s*:\s*([^\s#]+)", text)
    return sorted({n for n in names if n != "*"}, key=str.lower)

def check(text, base, paths):
    rp = Protego.parse(text)
    agents = named_agents(text) + ["SomeUnnamedBot"]  # last one exercises the * group
    leaks = []
    for ua in agents:
        for p in paths:
            ok = rp.can_fetch(base + p, ua)
            print(f"{ua:20s} {p:28s} {'allow' if ok else 'DISALLOW'}")
            if ok:
                leaks.append((ua, p))
    return leaks

if __name__ == "__main__":
    robots_file, base, *paths = sys.argv[1:]
    leaks = check(open(robots_file, encoding="utf-8").read(), base, paths)
    if leaks:
        print(f"\n{len(leaks)} blocked path(s) are still allowed")
        sys.exit(1)
python check_groups.py public/robots.txt https://example.com /go/amazon/B0CL6H35DY
```

On my old file, all 13 named groups came back allow and only `SomeUnnamedBot` was blocked, which is exactly what the logs showed. On the fixed file every row says DISALLOW and it exits 0. Put the paths you care about in CI and a new group without the Disallow will fail the build.

Python's standard `urllib.robotparser` says everything is allowed, even with the fixed file:

``` python
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.parse(open("robots.txt", encoding="utf-8").read().splitlines())
rp.can_fetch("Googlebot", "/go/amazon/B0CL6H35DY")  # True, even after the fix
```

It reads a group top to bottom and stops at the first matching line. With `Allow: /` first, it never reaches `Disallow: /go/`. Use [Protego](https://github.com/scrapy/protego), the longest-match parser Scrapy uses. I tested with 0.7.0.

My redirect links had `rel="sponsored nofollow"` and returned `X-Robots-Tag: noindex, nofollow`. Googlebot still fetched 67 different redirect URLs.

`X-Robots-Tag` comes back with the response, so by the time a crawler reads it, the request has already been made.
If you need the request itself to stop, use robots.txt.

The hour-by-hour data after the fix, why I can't credit the fix for ClaudeBot (it stopped two days earlier), and the unnamed GoogleOther that showed up anyway are on Aulvem → [Aulvem | User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times](https://aulvem.com/blog/2026-11-02-robots-txt-named-groups-ignore-wildcard/)