Cloudflare Workers AI Leaked Other Customers' Data Through GLM-5.3: 155 Responses, One SGLang Bug

Cloudflare disclosed that its hosted GLM-5.3 model on Workers AI returned other customers' data in 155 responses between August 27 and September 9, 2026, all to prompts longer than 681,280 tokens. Cloudflare attributed the cross-tenant leak to a mismatch between the model's 1,048,576-token context window and a 681,280-token prefill KV pool, combined with an SGLang bug that let rejected over-length requests proceed and attend over leftover KV cache pages; the pool was resized and SGLang patched in v0.5.20 and later. Gadi Evron of Knostic reported the issue through Cloudflare's HackerOne program on September 8, and Cloudflare said it found no evidence of deliberate exploitation and that no other Workers AI model was affected.

Cloudflare has disclosed that its hosted GLM-5.3 model on Workers AI sometimes returned data that belonged to other customers. According to Cloudflare's incident write-up https://blog.cloudflare.com/kv-cache-exposure-remediation/?ref=explainx , 155 responses between August 27 and September 9, 2026 were affected, all of them replies to prompts longer than 681,280 tokens. The cause was a configuration mismatch combined with a bug in the open-source SGLang inference engine, and both are now fixed. No other Workers AI model was affected, and Cloudflare says it found no sign of deliberate exploitation. This is a rare, detailed look at what a cross-tenant leak in LLM serving actually looks like. It is also a useful reminder for anyone who runs open-weight models https://explainx.ai/blog/glm-5-3-open-weights-delay-zai-august-2026 on shared GPUs. Below: what happened, the timeline, why the bug slipped through, what HN readers made of it, and a checklist for operators. Agents that act on your behalf need rules. Agentbeam stops the dangerous actions → https://app.agentbeam.com/login TL;DR: the Workers AI KV cache exposure | Question | Answer per Cloudflare | |---|---| | Which model? | GLM-5.3, @cf/zai-org/glm-5.3 , on Workers AI | | When? | August 27 to September 9, 2026 | | How many responses? | 155, all to prompts above 681,280 tokens | | How many customers? | "A small number"; no exact count given | | Root cause | Context window of 1,048,576 tokens vs a prefill KV pool of 681,280 tokens, plus an SGLang bug that let rejected requests proceed | | Who found it? | Gadi Evron of Knostic, via Cloudflare's HackerOne program, on September 8 | | Exploited on purpose? | No evidence found | | Fixed? | Yes: pool resized, SGLang patched v0.5.20 and later , more hardening shipped | What happened, in plain terms When a language model reads a prompt, the serving engine stores intermediate results called the KV cache so it does not recompute them for every new token. In large deployments the work is often split: a prefill node reads the prompt and builds the cache, then hands it to a decode node that generates the answer. This is called disaggregated prefill and decode, and it is common in engines such as SGLang. If you want background on why this cache matters so much for cost and speed, our explainer on DeepSeek's KV cache reduction https://explainx.ai/blog/deepseek-v4-1-flash-kv-cache-hbm-reduction-september-2026 covers the economics. Cloudflare launched GLM-5.3 on August 27 configured with a context window of 1,048,576 tokens. But the memory pool on its prefill GPUs only had room for 681,280 tokens. In Cloudflare's words, the team was aware of the mismatch before launch. A prompt longer than the pool should have been rejected with an error. That is where the SGLang bug comes in. In disaggregated mode, an over-length prompt was marked as rejected and swapped for a one-token placeholder. The queues that admit requests checked whether a request was finished, not whether it was rejected, so the request carried on. The decode node then attended over KV cache pages left behind by earlier requests and generated text from them. The customer got an HTTP 200 with another person's context woven into the answer, instead of an error. A bead held in a side pocket, illustrating how a KV cache stores data that a Workers AI KV cache leak exposed to the wrong request The bug itself was not new. Cloudflare says it was reported upstream as SGLang issue 30233 on July 6, 2026, weeks before the Workers AI launch. The write-up does not suggest anyone at Cloudflare had connected that issue to their own deployment before the report arrived. Timeline UTC | Date and time | Event | |---|---| | Aug 27 | GLM-5.3 launches on Workers AI with the undersized prefill pool | | Sep 8, 23:11 | Report received through HackerOne | | Sep 9, 13:22 | Security incident declared; Product Security reproduces the leak | | Sep 9, 14:00 | Model set to private; requests return HTTP 400 | | Sep 9, 14:35 | Root cause identified | | Sep 9, 19:04 | Corrected deployment with engine patch built and validated | | Sep 9, 19:40 | Model restored to public | | Sep 10 | Meeting with SGLang maintainers at RadixArk | | Sep 11 to 19 | Extra hardening rolled out to production SGLang deployments | | Oct 2 | SGLang v0.5.21 released | | Oct 19 | SGLang v0.5.22 released | | Oct 26 | Coordinated public disclosure | Two things stand out. First, the response was fast: about 14 hours from the report to a declared incident and a few hours more to a fix. Second, public disclosure came roughly seven weeks later, after the upstream fixes had shipped. That is a normal coordinated-disclosure pattern, but it does mean customers learned about the exposure well after the window closed. Why Cloudflare's testing missed it The most instructive part of the post is the honest root-cause note. Cloudflare says its pre-launch tests checked HTTP status codes and streaming behavior, not whether the output was correct for very long prompts. A leaked answer still returned a clean 200 and a plausible stream, so every dashboard stayed green. That is a general lesson for inference teams. Correctness tests for long-context serving need canary markers: unique strings planted in one request that must never show up in another. Cloudflare says it now validates fixes against the researcher's reproduction script and its own canary-marker tests, and has added pre-deployment checks that compare KV pool size with advertised context length. It also echoes a pattern we have seen on this site before: speed and capacity claims for new models are easy to verify at launch, while isolation properties are not. Our earlier coverage of Cloudflare's Clairic model on Workers AI https://explainx.ai/blog/cloudflare-clairic-system-one-extraction-model-workers-ai-claimed-vs-verified-2026 looked at what can and cannot be checked from outside. What was exposed, and what was not Cloudflare's post gives the count 155 responses and the trigger prompts above 681,280 tokens , but it does not describe the content of the leaked fragments, and it does not say how many distinct customers were involved beyond "a small number." We therefore cannot say what kinds of data, whether credentials, source code or personal information, appeared in those replies. What the post does say: the affected requests matched the timing and volume of ordinary long-context use, which is why Cloudflare concluded there was no evidence of deliberate exploitation. An attacker would have needed to send over-length prompts and then hope to read useful leftovers, which is a lottery rather than a targeted attack. That is reassuring but not a guarantee, and it is Cloudflare's own assessment, not an independent audit. Customers who sent very long prompts to GLM-5.3 during the window should look at their own logs for anything unexpected in responses. The post does not list specific steps for Workers AI customers beyond reporting security issues at Cloudflare's HackerOne program, so contact your account team if you think you were affected. The fix, and what it changes Cloudflare describes four layers of remediation: 1. Configuration. The GLM-5.3 prefill pool was resized to about 2.04 million tokens, roughly twice the context length, so a valid request always fits. 2. Engine fix. Rejected requests are now retired with their original error before any KV transfer happens. That fix shipped upstream in SGLang v0.5.20 and later. 3. Hardening. Deferred KV release, bounded Mooncake transfer batches, load balancer changes to avoid mid-transfer aborts, and pre-deployment checks of pool size against context length. 4. Validation. Tests using the researcher's script and canary markers. For the future, Cloudflare says it plans page-level integrity tags in the engine so that a read of a page belonging to another request fails instead of returning data, and it is applying formal verification to its RDMA-based KV transfer protocol. Those are plans, not shipped features. What developers are saying The Hacker News thread had 36 points and eight comments when we read it, so it is a small sample. Still, a few reactions are worth noting, all presented as the commenters' own views. - User kami23 praised the write-up https://news.ycombinator.com/item?id=50252432&ref=explainx , saying they liked this root-cause analysis more than most for its ownership and its explanation of why testing missed the bug, and noted that taking the model offline quickly was good to see. - User 0xbadcafebee said they had half-forgotten the shared-tenancy attack model https://news.ycombinator.com/item?id=50252800&ref=explainx , had been hoping software magically isolated their tokens, and argued enterprises with highly sensitive data should use non-shared hosts. - User segmondy questioned the launch decision https://news.ycombinator.com/item?id=50252927&ref=explainx : did Cloudflare offer a 1,048,576-token window when the hardware only had room for 681,280? The post itself says the team knew about the mismatch before launch. - User thehamkercat guessed other providers may not know or care https://news.ycombinator.com/item?id=50252267&ref=explainx that similar things are happening, and user 8cvor6j844qw d6 wondered whether this explained https://news.ycombinator.com/item?id=50252438&ref=explainx unrelated responses they had seen from another provider's API. Both are speculation; nothing in Cloudflare's post links this bug to any other provider. What this means if you run SGLang or use hosted inference If you operate your own SGLang cluster, Cloudflare's checklist is concrete and worth running this week: 1. After any configuration change, compare the KV pool size on every prefill and decode node with the model's context length. 2. Upgrade to SGLang v0.5.22 or later. Our look at the engine's plugin system and vendor kernels https://explainx.ai/blog/sglang-plugin-system-sgl-nvidia-mega-sgl-amd-mono-vendor-kernels-2026 is useful background on how fast the project is moving. 3. Enable deferred KV release on both roles, and bound Mooncake transfer batches. 4. Turn on corruption detection where the model supports it. 5. Configure proxies so they do not abort requests mid-transfer. 6. Add a canary test: send a unique marker in one session and assert it never appears in any other session's output, including at maximum context length. If you are a customer of a hosted model, the practical advice is more modest but real: - Do not put secrets in long prompts unless you need to, and treat any shared-tenant API as a boundary that can fail. - Scan outputs for strings that should not be there, especially in long-context jobs such as whole-repository analysis. - Pick the deployment type to match the data. Providers differ on dedicated capacity, confidential-computing options and logging. For example, Brave built an end-to-end encrypted chat on GLM-5.3 Flash using trusted execution environments https://explainx.ai/blog/brave-leo-end-to-end-encrypted-ai-chat-glm-5-3-flash-near-ai-tee-ohttp-2026 , which addresses a different threat but shows the direction. - Run agents with least privilege. A leak that reveals another user's tokens is far worse when an agent holds broad credentials. AgentBeam https://agentbeam.com/?ref=explainx , the agent security platform from the explainx.ai team, is built to stop AI agents before they take dangerous actions, which limits what a stray credential can do. Why GLM-5.3 specifically, and is the model at fault? Nothing in the disclosure blames the model. GLM-5.3 is Z.ai's open-weight model, which we covered at launch for its cyber-defense benchmarks https://explainx.ai/blog/glm-5-3-launch-cyber-defense-benchmarks-august-2026 and which is now available on several clouds, including Amazon Bedrock https://explainx.ai/blog/glm-5-3-amazon-bedrock-enterprise-coding-model-ids-2026 . The failure sat entirely in the serving layer: a deployment setting and an inference-engine bug. Other providers running the same weights on different stacks are not implicated by this report, though operators using SGLang in disaggregated mode with a similar mismatch should check their own setups. Claimed vs verified | Claim | Status | |---|---| | 155 responses leaked other customers' data | Cloudflare's own count; not independently audited | | Only prompts above 681,280 tokens triggered it | Cloudflare's finding; consistent with the technical cause described | | No evidence of deliberate exploitation | Cloudflare's assessment | | The SGLang bug was reported upstream on July 6 | Cited by Cloudflare as SGLang issue 30233 | | Fixes shipped in SGLang v0.5.20 and later | Per Cloudflare; check the SGLang releases page https://github.com/sgl-project/sglang/releases?ref=explainx for your version | | Content of leaked fragments | Not disclosed | Bottom line The incident is small in count and large in lesson. A model that is "up" and returning 200s can still be leaking, and the two gaps that allowed it, a known context-window mismatch and a known upstream bug, were each visible before launch. Cloudflare's candid write-up, fast mitigation and concrete operator checklist are what a good post-mortem looks like, but customers are left to trust the numbers. Related reading: Details here follow Cloudflare's October 26, 2026 disclosure and the Hacker News thread at the time of writing; versions and fixes may change.