Our own CDN was 403-ing the official OpenAI SDK, and our test suite couldn't see it

A developer's bot-blocking rule at the CDN edge mistakenly returned 403 errors to the official OpenAI and Anthropic SDKs, causing an outage that went undetected because the test suite's probe used an allowlisted User-Agent. The developer identified three mistakes and provided a method to detect such UA-based filtering by comparing responses with and without the SDK's default User-Agent.

We wrote a bot-blocking rule at the CDN edge. It matched the official OpenAI and Anthropic SDKs by the shape of their User-Agent string and returned 403 to all of them. It ran that way for a while, and our compatibility test suite reported green the entire time. The suite was green because the probe sent its own User-Agent, and its own User-Agent was on the allowlist. That is the part worth your time. The outage is ordinary; the blind spot is structural, and it is sitting in most home-grown test tooling I have seen. Three mistakes below, all ours. Numbers are from the retest on 2026-08-31. The rule was aimed at AI crawlers. It matched on a Vendor/Language version User-Agent pattern, which is exactly what the official SDKs send by default: OpenAI/Python 1.68.2 OpenAI/JS 4.104.0 Anthropic/JS 0.120.0 The damage split into two layers, which is why it stayed invisible for so long: 403 Your request was blocked. before their request reached the application at all. llms.txt assets we had written specifically for them were unreachable.The 403 is worse than a plain outage, because of what it looks like from the other side. A blocked request returns 403, not 401. A developer whose first-ever request to your API returns 403 concludes that they pasted the key wrong. They will not file a bug. They will close the tab. We do not install nine GUI clients and click through them. We reproduce the characteristic HTTP request each client sends, fire it at the real user-facing endpoint, and check the response field by field. It is repeatable, and repeatable matters when the upstream model catalog changes under you. That suite was green. It tested protocol shape: streaming frames, tool-call assembly, response format , the Anthropic event sequence. Every one of those answers the question does the server understand this request . Not one of them answers does this request get in . We found it by accident, while double-checking a streaming result with the official Anthropic SDK. That call returned 403 Your request was blocked. The identical request sent with curl returned 200. The only difference between the two was the User-Agent header. Our probe's UA was in the allowed set. So the suite was structurally incapable of seeing a rule that keyed on UA. It was not a missing test case; it was a test client whose identity happened to be the safe one. Two requests, same body, same path, same key. Only the User-Agent differs. If the statuses disagree, something in front of your application is filtering on UA. 1. Baseline. curl's default UA is not an SDK shape. curl -sS -o /dev/null -w '%{http code}\n' \ -X POST "$BASE URL/chat/completions" \ -H "authorization: Bearer $API KEY" \ -H 'content-type: application/json' \ -d '{"model":"'"$MODEL"'","messages": {"role":"user","content":"hi"} ,"max tokens":4}' 2. Same request, wearing the official SDK's default UA. curl -sS -o /dev/null -w '%{http code}\n' \ -X POST "$BASE URL/chat/completions" \ -H "authorization: Bearer $API KEY" \ -H 'content-type: application/json' \ -H 'user-agent: OpenAI/Python 1.68.2' \ -d '{"model":"'"$MODEL"'","messages": {"role":"user","content":"hi"} ,"max tokens":4}' Then confirm with the real SDK, because a hand-written header is still your guess at what the SDK sends: python from openai import OpenAI client = OpenAI base url=BASE URL, api key=API KEY print client.chat.completions.create model=MODEL, messages= {"role": "user", "content": "hi"} , max tokens=4, .choices 0 .message.content Do the same for crawlers, against a static asset rather than the API: curl -sS -o /dev/null -w '%{http code}\n' \ -H 'user-agent: Mozilla/5.0 AppleWebKit/537.36 KHTML, like Gecko ; compatible; GPTBot/1.2; +https://openai.com/gptbot' \ https://your-site.example/llms.txt If you want a live target to check the harness against before pointing it at your own host, our /llms.txt https://y-api.bestvirtualgoods.com/llms.txt answers 200 to all Two details that make the check reliable. An edge block has a signature worth matching on rather than eyeballing: status 403, plus a server: cloudflare header, plus a very short text/plain body — we treat under 200 bytes as the cutoff. And run the unauthenticated version too: with no credentials, a healthy endpoint answers 401 Invalid token . If it answers 403 instead, you never reached the application. The fix was in the Cloudflare rule. No application code changed. That is worth stating plainly, because for the entire time this was broken, every line of our request-handling code was correct and every protocol probe agreed. The retest on 2026-08-31: llms.txt .The 6 were the ones that send an SDK-shaped UA: Cline, Roo Code, LibreChat, Continue, Claude Code, and the official OpenAI SDK. Aider and Open WebUI were never blocked, because neither sends that shape — Aider goes through LiteLLM litellm/1.63.0 , lowercase, hyphenated and Open WebUI uses aiohttp directly. Two clients passing by accident is not coverage. The 9th row is still not usable, and not for reachability reasons. It is the official OpenAI SDK, which we keep as a strict baseline: it requires vision and a well-formed error shape, and we fail both. Our vision column is worse than the reachability story suggests — we sent five upstream models a solid green PNG and asked what color it was, and all five got it wrong, so we do not claim vision support. Unknown model names return 503 instead of 4xx, which makes SDKs retry a typo as if it were a server fault. Those are open, not fixed. We also added the missing probes: a reachability gate that replays each client's real UA, run before anything else. When that gate is red the protocol results below it are meaningless, so it now runs first. The suite's primary model was hard-coded to the model ID in our documentation examples. When the upstream catalog dropped that ID, the probe threw on startup and all 19 checks refused to run. A recoverable drift became a hard block, because a single model name was wired up as the master switch. The catalog gets reshuffled routinely and there is always some working model, so this was the wrong failure mode. It now picks dynamically: llms-full.txt , so when it stops being callable, every copy-paste path breaks at step one and no page on the site looks any different.That third point is the actual repair. The check was worth keeping; it just should never have been able to stop everything else from running. This one took the longest to notice, and it is the one I would most expect other people to have shipped. The compatibility matrix is generated. The script overwrites the file on every run, and that file is the single source of truth for everything we say publicly about what we support. We ran it once while the upstream was degraded. A non-streaming request took 89 seconds against a 120-second timeout — slow enough to be useless, fast enough to not time out. Requests that did cross 120 seconds, plus some intermittent 404s, were recorded as genuine failures. The generator did what it was told and wrote a document marking all nine clients unusable , including Aider and Open WebUI, which were working the whole time. For as long as that file stood, our own documentation claimed we supported nothing. Nobody had to be wrong for this to happen; the script was correct and the output was false. The fix is a health gate. The script now measures the primary model's latency, and past a 30-second budget it declares the run degraded, refuses to write the file, and prints results to stdout for a human to read. A stale document beats a confidently wrong one, because a stale one is at least a claim somebody once verified. If you generate anything user-facing from live measurements, this applies to you directly: a generator with write access and no health gate will eventually publish your worst five minutes as your steady state. Reproducing a client's request shape is a good technique and I would use it again. It is repeatable, it is cheap, and it catches real protocol breakage. But it tests the request. It does not test the requester. Everything that gets decided by who is asking — UA rules, bot detection, WAF heuristics, rate-limit buckets, IP reputation — is invisible to a probe that shows up wearing its own identity. And a home-grown probe always shows up wearing an identity you chose, which means it is almost always in the group you decided to let through. That is not a gap you can close by adding assertions. You close it by sending the request as the client, with the client's real User-Agent, or by installing the actual SDK and letting it speak for itself. The question to ask about your own test suite is not "what does it check." It is "which class of client is it, and who else is in that class." Ours was in the class we had allowlisted, so the 403 was never something it could have reported.