We wrote a bot-blocking rule at the CDN edge. It matched the official OpenAI and
Anthropic SDKs by the shape of their User-Agent string and returned 403 to all of
them. It ran that way for a while, and our compatibility test suite reported green
the entire time.
The suite was green because the probe sent its own User-Agent, and its own
User-Agent was on the allowlist. That is the part worth your time. The outage is
ordinary; the blind spot is structural, and it is sitting in most home-grown test
tooling I have seen.
Three mistakes below, all ours. Numbers are from the retest on 2026-08-31.
The rule was aimed at AI crawlers. It matched on a Vendor/Language version
User-Agent pattern, which is exactly what the official SDKs send by default:
OpenAI/Python 1.68.2
OpenAI/JS 4.104.0
Anthropic/JS 0.120.0
The damage split into two layers, which is why it stayed invisible for so long:
403 Your request was blocked.
before their request reached the
application at all.llms.txt
assets we had written specifically for them were unreachable.The 403 is worse than a plain outage, because of what it looks like from the other
side. A blocked request returns 403, not 401. A developer whose first-ever request
to your API returns 403 concludes that they pasted the key wrong. They will not file
a bug. They will close the tab.
We do not install nine GUI clients and click through them. We reproduce the
characteristic HTTP request each client sends, fire it at the real user-facing
endpoint, and check the response field by field. It is repeatable, and repeatable
matters when the upstream model catalog changes under you.
That suite was green. It tested protocol shape: streaming frames, tool-call
assembly, response_format
, the Anthropic event sequence. Every one of those
answers the question does the server understand this request. Not one of them
answers does this request get in.
We found it by accident, while double-checking a streaming result with the official
Anthropic SDK. That call returned 403 Your request was blocked.
The identical
request sent with curl returned 200. The only difference between the two was the
User-Agent header.
Our probe's UA was in the allowed set. So the suite was structurally incapable of
seeing a rule that keyed on UA. It was not a missing test case; it was a test client
whose identity happened to be the safe one.
Two requests, same body, same path, same key. Only the User-Agent differs. If the
statuses disagree, something in front of your application is filtering on UA.
curl -sS -o /dev/null -w '%{http_code}\n' \
-X POST "$BASE_URL/chat/completions" \
-H "authorization: Bearer $API_KEY" \
-H 'content-type: application/json' \
-d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"hi"}],"max_tokens":4}'
curl -sS -o /dev/null -w '%{http_code}\n' \
-X POST "$BASE_URL/chat/completions" \
-H "authorization: Bearer $API_KEY" \
-H 'content-type: application/json' \
-H 'user-agent: OpenAI/Python 1.68.2' \
-d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"hi"}],"max_tokens":4}'
Then confirm with the real SDK, because a hand-written header is still your guess at
what the SDK sends:
from openai import OpenAI
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
print(client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "hi"}],
max_tokens=4,
).choices[0].message.content)
Do the same for crawlers, against a static asset rather than the API:
curl -sS -o /dev/null -w '%{http_code}\n' \
-H 'user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot' \
https://your-site.example/llms.txt
If you want a live target to check the harness against before pointing it at your own
host, our /llms.txt answers 200 to all
Two details that make the check reliable. An edge block has a signature worth
matching on rather than eyeballing: status 403, plus a server: cloudflare
header,
plus a very short text/plain
body — we treat under 200 bytes as the cutoff.
And run the unauthenticated version too: with no credentials, a healthy endpoint
answers 401 Invalid token
. If it answers 403 instead, you never reached the
application.
The fix was in the Cloudflare rule. No application code changed. That is worth
stating plainly, because for the entire time this was broken, every line of our
request-handling code was correct and every protocol probe agreed.
The retest on 2026-08-31:
llms.txt
.The 6 were the ones that send an SDK-shaped UA: Cline, Roo Code, LibreChat,
Continue, Claude Code, and the official OpenAI SDK. Aider and Open WebUI were never
blocked, because neither sends that shape — Aider goes through LiteLLM
(litellm/1.63.0
, lowercase, hyphenated) and Open WebUI uses aiohttp directly. Two
clients passing by accident is not coverage.
The 9th row is still not usable, and not for reachability reasons. It is the official
OpenAI SDK, which we keep as a strict baseline: it requires vision and a
well-formed error shape, and we fail both. Our vision column is worse than the
reachability story suggests — we sent five upstream models a solid green PNG and
asked what color it was, and all five got it wrong, so we do not claim vision
support. Unknown model names return 503 instead of 4xx, which makes SDKs retry a
typo as if it were a server fault. Those are open, not fixed.
We also added the missing probes: a reachability gate that replays each client's
real UA, run before anything else. When that gate is red the protocol results below
it are meaningless, so it now runs first.
The suite's primary model was hard-coded to the model ID in our documentation
examples. When the upstream catalog dropped that ID, the probe threw on startup and
all 19 checks refused to run. A recoverable drift became a hard block, because a
single model name was wired up as the master switch.
The catalog gets reshuffled routinely and there is always some working model, so
this was the wrong failure mode. It now picks dynamically:
llms-full.txt
, so when it stops being callable, every copy-paste path breaks at step one and no page on the site looks any different.That third point is the actual repair. The check was worth keeping; it just should
never have been able to stop everything else from running.
This one took the longest to notice, and it is the one I would most expect other
people to have shipped.
The compatibility matrix is generated. The script overwrites the file on every run,
and that file is the single source of truth for everything we say publicly about
what we support.
We ran it once while the upstream was degraded. A non-streaming request took 89
seconds against a 120-second timeout — slow enough to be useless, fast enough to not
time out. Requests that did cross 120 seconds, plus some intermittent 404s, were
recorded as genuine failures. The generator did what it was told and wrote a
document marking all nine clients unusable, including Aider and Open WebUI,
which were working the whole time.
For as long as that file stood, our own documentation claimed we supported nothing.
Nobody had to be wrong for this to happen; the script was correct and the output was
false.
The fix is a health gate. The script now measures the primary model's latency, and
past a 30-second budget it declares the run degraded, refuses to write the file, and
prints results to stdout for a human to read. A stale document beats a confidently
wrong one, because a stale one is at least a claim somebody once verified.
If you generate anything user-facing from live measurements, this applies to you
directly: a generator with write access and no health gate will eventually publish
your worst five minutes as your steady state.
Reproducing a client's request shape is a good technique and I would use it again.
It is repeatable, it is cheap, and it catches real protocol breakage.
But it tests the request. It does not test the requester. Everything that gets
decided by who is asking — UA rules, bot detection, WAF heuristics, rate-limit
buckets, IP reputation — is invisible to a probe that shows up wearing its own
identity. And a home-grown probe always shows up wearing an identity you chose,
which means it is almost always in the group you decided to let through. That is not
a gap you can close by adding assertions. You close it by sending the request as the
client, with the client's real User-Agent, or by installing the actual SDK and
letting it speak for itself.
The question to ask about your own test suite is not "what does it check." It is
"which class of client is it, and who else is in that class." Ours was in the class
we had allowlisted, so the 403 was never something it could have reported.