Cloudflare: GLM-5.3 Bug Leaked Data Across Customers

Cloudflare has disclosed a cross-customer data exposure in Workers AI, its serverless inference service. For almost two weeks, from 27 August to 9 September, the GLM-5.3 model it hosts (@cf/zai-org/glm-5.3, Z.ai’s agentic coding model) sometimes answered one customer’s request with text drawn from another customer’s recent request. Cloudflare counts 155 affected responses, all to prompts longer than 681,280 tokens, and says no other Workers AI model was affected. After excluding its own testing and the researcher’s, it found “no evidence of deliberate exploitation”: the affected requests looked like ordinary long-context use in timing and volume. The post, written by Michelle Chen and published on 26 October, was coordinated with Gadi Evron of Knostic, who reported the bug, and with the SGLang team, whose serving engine was at fault.

Two problems overlapped. The first was a configuration mismatch. Cloudflare advertised GLM-5.3 with a 1,048,576-token context window, but the KV cache on its prefill GPUs, the memory that holds a prompt’s processed state, only had room for 681,280 tokens. SGLang sizes that pool at startup from whatever GPU memory is left after loading the weights, and nothing forces it to match the context length. Cloudflare knew about the gap before launch and expected oversized prompts to fail with an error. Its long-context tests came back successful, so nobody treated it as a problem; those tests checked status codes and speed, not whether the output was correct.

The second was a bug in SGLang’s disaggregated mode, where one set of GPUs processes the prompt (prefill) and another generates the reply (decode). When a prompt did not fit, SGLang marked it rejected and swapped in a one-token placeholder. On a single node that returns an error. Split across nodes, both admission queues checked whether a request was finished rather than whether it was rejected, so it went through. In Cloudflare’s example, decode reserved eight pages of KV cache for the full prompt, prefill wrote only the first, and decode read all eight. Pages are not wiped between requests, so the other seven still held earlier users’ data. The client got an HTTP 200, the correct prompt-token count and a fluent reply built partly on someone else’s session. The underlying bug had been reported upstream as SGLang issue #30233 on 6 July.

The report came in through Cloudflare’s HackerOne programme at 23:11 UTC on 8 September. Cloudflare declared an incident at 13:22 UTC the next day, its product security team reproduced the leak with two separate accounts within 25 minutes, and the model went private at 14:00. It was back online at 19:40 UTC the same day with two fixes: a prefill pool of about 2.04 million tokens, roughly twice the context length, and an engine patch, proposed upstream as PR #38704, that retires a rejected request with its original error before any memory is allocated or transferred. Further hardening followed across all of Cloudflare’s SGLang deployments, and every change to a disaggregated deployment now has to pass checks on pool size against context length.

The SGLang post widens the story beyond one provider. While investigating, the maintainers found a related race: if a request is aborted mid-transfer, decode can free its memory slots while asynchronous writes are still landing, and those late writes can overwrite another request’s cache, silently and without a log line. SGLang says default configurations were exposed from the v0.4.x line in early 2025 through v0.5.21, or v0.5.22 for the MORI transfer backend, and that the same steps apply to “any PD implementation” that moves KV cache with asynchronous DMA. Its advice is to upgrade (v0.5.21 or later for Mooncake and NIXL; Cloudflare recommends v0.5.22), run a router that cancels both halves of an aborted request, and consider the optional KV checksum. The fixes cost nothing measurable: across 21 runs on two H100s, throughput moved by less than 2%.

BitsMinds has followed GLM-5.3 since August, and the model itself did nothing wrong here. The failure sat in the serving layer, the kind of infrastructure most customers never see and simply trust a provider to get right. Cloudflare’s own conclusion is that tests have to check what a model says, not just that it answered, and that is now the lesson for every host running a split prefill/decode setup, not only Cloudflare.