{"slug": "our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldn-t-see", "title": "Our own CDN was 403-ing the official OpenAI SDK, and our test suite couldn't see it", "summary": "A developer's bot-blocking rule at the CDN edge mistakenly returned 403 errors to the official OpenAI and Anthropic SDKs, causing an outage that went undetected because the test suite's probe used an allowlisted User-Agent. The developer identified three mistakes and provided a method to detect such UA-based filtering by comparing responses with and without the SDK's default User-Agent.", "body_md": "We wrote a bot-blocking rule at the CDN edge. It matched the official OpenAI and\n\nAnthropic SDKs by the shape of their User-Agent string and returned 403 to all of\n\nthem. It ran that way for a while, and our compatibility test suite reported green\n\nthe entire time.\n\nThe suite was green because the probe sent its own User-Agent, and its own\n\nUser-Agent was on the allowlist. That is the part worth your time. The outage is\n\nordinary; the blind spot is structural, and it is sitting in most home-grown test\n\ntooling I have seen.\n\nThree mistakes below, all ours. Numbers are from the retest on 2026-08-31.\n\nThe rule was aimed at AI crawlers. It matched on a `Vendor/Language version`\n\nUser-Agent pattern, which is exactly what the official SDKs send by default:\n\n`OpenAI/Python 1.68.2`\n\n`OpenAI/JS 4.104.0`\n\n`Anthropic/JS 0.120.0`\n\nThe damage split into two layers, which is why it stayed invisible for so long:\n\n`403 Your request was blocked.`\n\nbefore their request reached the\napplication at all.`llms.txt`\n\nassets we had written specifically for them were unreachable.The 403 is worse than a plain outage, because of what it looks like from the other\n\nside. A blocked request returns 403, not 401. A developer whose first-ever request\n\nto your API returns 403 concludes that they pasted the key wrong. They will not file\n\na bug. They will close the tab.\n\nWe do not install nine GUI clients and click through them. We reproduce the\n\ncharacteristic HTTP request each client sends, fire it at the real user-facing\n\nendpoint, and check the response field by field. It is repeatable, and repeatable\n\nmatters when the upstream model catalog changes under you.\n\nThat suite was green. It tested protocol shape: streaming frames, tool-call\n\nassembly, `response_format`\n\n, the Anthropic event sequence. Every one of those\n\nanswers the question *does the server understand this request*. Not one of them\n\nanswers *does this request get in*.\n\nWe found it by accident, while double-checking a streaming result with the official\n\nAnthropic SDK. That call returned `403 Your request was blocked.`\n\nThe identical\n\nrequest sent with curl returned 200. The only difference between the two was the\n\nUser-Agent header.\n\nOur probe's UA was in the allowed set. So the suite was structurally incapable of\n\nseeing a rule that keyed on UA. It was not a missing test case; it was a test client\n\nwhose identity happened to be the safe one.\n\nTwo requests, same body, same path, same key. Only the User-Agent differs. If the\n\nstatuses disagree, something in front of your application is filtering on UA.\n\n```\n# 1. Baseline. curl's default UA is not an SDK shape.\ncurl -sS -o /dev/null -w '%{http_code}\\n' \\\n  -X POST \"$BASE_URL/chat/completions\" \\\n  -H \"authorization: Bearer $API_KEY\" \\\n  -H 'content-type: application/json' \\\n  -d '{\"model\":\"'\"$MODEL\"'\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":4}'\n\n# 2. Same request, wearing the official SDK's default UA.\ncurl -sS -o /dev/null -w '%{http_code}\\n' \\\n  -X POST \"$BASE_URL/chat/completions\" \\\n  -H \"authorization: Bearer $API_KEY\" \\\n  -H 'content-type: application/json' \\\n  -H 'user-agent: OpenAI/Python 1.68.2' \\\n  -d '{\"model\":\"'\"$MODEL\"'\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":4}'\n```\n\nThen confirm with the real SDK, because a hand-written header is still your guess at\n\nwhat the SDK sends:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=BASE_URL, api_key=API_KEY)\nprint(client.chat.completions.create(\n    model=MODEL,\n    messages=[{\"role\": \"user\", \"content\": \"hi\"}],\n    max_tokens=4,\n).choices[0].message.content)\n```\n\nDo the same for crawlers, against a static asset rather than the API:\n\n```\ncurl -sS -o /dev/null -w '%{http_code}\\n' \\\n  -H 'user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot' \\\n  https://your-site.example/llms.txt\n```\n\nIf you want a live target to check the harness against before pointing it at your own\n\nhost, our [ /llms.txt](https://y-api.bestvirtualgoods.com/llms.txt) answers 200 to all\n\nTwo details that make the check reliable. An edge block has a signature worth\n\nmatching on rather than eyeballing: status 403, plus a `server: cloudflare`\n\nheader,\n\nplus a very short `text/plain`\n\nbody — we treat under 200 bytes as the cutoff.\n\nAnd run the unauthenticated version too: with no credentials, a healthy endpoint\n\nanswers 401 `Invalid token`\n\n. If it answers 403 instead, you never reached the\n\napplication.\n\nThe fix was in the Cloudflare rule. No application code changed. That is worth\n\nstating plainly, because for the entire time this was broken, every line of our\n\nrequest-handling code was correct and every protocol probe agreed.\n\nThe retest on 2026-08-31:\n\n`llms.txt`\n\n.The 6 were the ones that send an SDK-shaped UA: Cline, Roo Code, LibreChat,\n\nContinue, Claude Code, and the official OpenAI SDK. Aider and Open WebUI were never\n\nblocked, because neither sends that shape — Aider goes through LiteLLM\n\n(`litellm/1.63.0`\n\n, lowercase, hyphenated) and Open WebUI uses aiohttp directly. Two\n\nclients passing by accident is not coverage.\n\nThe 9th row is still not usable, and not for reachability reasons. It is the official\n\nOpenAI SDK, which we keep as a strict baseline: it requires vision and a\n\nwell-formed error shape, and we fail both. Our vision column is worse than the\n\nreachability story suggests — we sent five upstream models a solid green PNG and\n\nasked what color it was, and all five got it wrong, so we do not claim vision\n\nsupport. Unknown model names return 503 instead of 4xx, which makes SDKs retry a\n\ntypo as if it were a server fault. Those are open, not fixed.\n\nWe also added the missing probes: a reachability gate that replays each client's\n\nreal UA, run before anything else. When that gate is red the protocol results below\n\nit are meaningless, so it now runs first.\n\nThe suite's primary model was hard-coded to the model ID in our documentation\n\nexamples. When the upstream catalog dropped that ID, the probe threw on startup and\n\nall 19 checks refused to run. A recoverable drift became a hard block, because a\n\nsingle model name was wired up as the master switch.\n\nThe catalog gets reshuffled routinely and there is always *some* working model, so\n\nthis was the wrong failure mode. It now picks dynamically:\n\n`llms-full.txt`\n\n, so when it\nstops being callable, every copy-paste path breaks at step one and no page on the\nsite looks any different.That third point is the actual repair. The check was worth keeping; it just should\n\nnever have been able to stop everything else from running.\n\nThis one took the longest to notice, and it is the one I would most expect other\n\npeople to have shipped.\n\nThe compatibility matrix is generated. The script overwrites the file on every run,\n\nand that file is the single source of truth for everything we say publicly about\n\nwhat we support.\n\nWe ran it once while the upstream was degraded. A non-streaming request took 89\n\nseconds against a 120-second timeout — slow enough to be useless, fast enough to not\n\ntime out. Requests that did cross 120 seconds, plus some intermittent 404s, were\n\nrecorded as genuine failures. The generator did what it was told and wrote a\n\ndocument marking **all nine clients unusable**, including Aider and Open WebUI,\n\nwhich were working the whole time.\n\nFor as long as that file stood, our own documentation claimed we supported nothing.\n\nNobody had to be wrong for this to happen; the script was correct and the output was\n\nfalse.\n\nThe fix is a health gate. The script now measures the primary model's latency, and\n\npast a 30-second budget it declares the run degraded, refuses to write the file, and\n\nprints results to stdout for a human to read. A stale document beats a confidently\n\nwrong one, because a stale one is at least a claim somebody once verified.\n\nIf you generate anything user-facing from live measurements, this applies to you\n\ndirectly: a generator with write access and no health gate will eventually publish\n\nyour worst five minutes as your steady state.\n\nReproducing a client's *request shape* is a good technique and I would use it again.\n\nIt is repeatable, it is cheap, and it catches real protocol breakage.\n\nBut it tests the request. It does not test the requester. Everything that gets\n\ndecided by who is asking — UA rules, bot detection, WAF heuristics, rate-limit\n\nbuckets, IP reputation — is invisible to a probe that shows up wearing its own\n\nidentity. And a home-grown probe always shows up wearing an identity you chose,\n\nwhich means it is almost always in the group you decided to let through. That is not\n\na gap you can close by adding assertions. You close it by sending the request as the\n\nclient, with the client's real User-Agent, or by installing the actual SDK and\n\nletting it speak for itself.\n\nThe question to ask about your own test suite is not \"what does it check.\" It is\n\n\"which class of client is it, and who else is in that class.\" Ours was in the class\n\nwe had allowlisted, so the 403 was never something it could have reported.", "url": "https://wpnews.pro/news/our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldn-t-see", "canonical_source": "https://dev.to/freeourdays/our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldnt-see-it-3lb7", "published_at": "2026-08-31 16:17:52+00:00", "updated_at": "2026-08-31 16:22:10.993168+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "Anthropic", "CDN"], "alternates": {"html": "https://wpnews.pro/news/our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldn-t-see", "markdown": "https://wpnews.pro/news/our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldn-t-see.md", "text": "https://wpnews.pro/news/our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldn-t-see.txt", "jsonld": "https://wpnews.pro/news/our-own-cdn-was-403-ing-the-official-openai-sdk-and-our-test-suite-couldn-t-see.jsonld"}}