Cross-request KV cache publicity within the GLM-5.3 Workers AI mannequin, and our remediation with SGLang

On September 8, 2026, we had been alerted by a safety researcher, Gadi Evron from Knostic, to a tenant isolation concern affecting the GLM-5.3 Workers AI mannequin (@cf/zai-org/glm-5.3) beginning on August 27, 2026. We have remediated the problem, and we restored the mannequin on September 9. We have confirmed that no different mannequin hosted on Workers AI was affected.

Between August 27 and September 9, a small variety of prospects utilizing the GLM 5.3 Workers AI mannequin acquired responses that contained knowledge from different prospects’ requests made to the identical mannequin shortly earlier than theirs. In complete, 155 responses had been affected, all to prompts longer than 681,280 tokens.

After excluding our personal testing and the researcher’s, the requests we recognized are constant in timing and quantity with unusual long-context use, not with somebody repeatedly exploiting the bug. We discovered no proof of deliberate exploitation.

Today we’re publishing this report alongside a publish from SGLang, the open-source inference engine that we use, as a coordinated disclosure. This publish describes the bug that prompted this concern, our response timeline, and the following hardening we’ve got carried out on our manufacturing SGLang deployments. We additionally element our continued efforts with the upstream maintainers of SGLang to handle this and related bugs.

What occurred

GLM is a textual content era mannequin from Z.ai that’s constructed for agentic coding workflows; it is among the many models obtainable by way of Workers AI. Gadi Evron submitted a report on September 8 by way of our HackerOne program stating that very lengthy prompts despatched to @cf/zai-org/glm-5.3 on Workers AI typically returned textual content that belonged to different customers’ periods. Cloudflare reproduced the problem the subsequent morning and took the mannequin offline inside 40 minutes of declaring an incident. We then deployed a corrected configuration and a patched inference engine, and restored service the identical day.  

This particular report of cross-request KV cache publicity was the results of two overlapping points. The first was a misconfiguration of the open-source inference engine, SGLang, utilized by Cloudflare. We deployed GLM 5.3 with a 1,048,576-token context window, however the KV cache pool on our prefill GPUs solely had room for 681,280 tokens. We had been conscious of this mismatch earlier than launch. We anticipated the engine to return an error for any immediate it couldn’t course of, and our long-context checks returned profitable responses, so we didn’t imagine this to be a difficulty.

The second was a bug in how SGLang dealt with the ensuing over-length requests in disaggregated prefill/decode mode. A immediate that didn’t match the pool was internally marked as rejected and changed with a one-token placeholder, however as a substitute of aborting the request and returning a shopper error, prefill generated the KV cache for the placeholder immediate and transferred it to the decode node. The decode node acquired a “switch full” message and assumed that prefill overwrote the KV cache from earlier requests with the total immediate for this request, when in reality it left the vast majority of the earlier state intact. The decode node subsequently generated a response whereas attending over stale cache pages that also held knowledge from earlier requests. As a consequence, responses to prompts longer than 681,280 tokens might comprise knowledge from different customers’ GLM 5.3 periods.

Immediate remediation

After disabling the model, we took the next actions to right this vulnerability earlier than restoring public entry to it:

To repair our configuration, we made the pool maintain the total context. We adjusted our GLM 5.3 configuration to match our GLM 5.2 configuration, which freed sufficient GPU reminiscence that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context size. A immediate that matches the context window now all the time suits the pool, so the rejection path is rarely reached for a sound request. This alone prevents the bug from being exercised.

To repair the engine, we additionally made the rejection path protected. Relying on pool dimension alone is fragile: any change to quantization, tensor parallelism, CUDA graph sizes, or a future mannequin model modifications the pool dimension in a means that is straightforward to overlook. Furthermore, there could also be different conditions during which the prefill scheduler can reject a request and set off the identical bug. To forestall this, we additionally patched the engine so {that a} request carrying a pending rejection is retired on the disaggregation admission queue, with its unique error, earlier than any bootstrap, allocation, or KV switch occurs. We proposed this upstream as #38704, which additionally had the decode node confirm that the variety of tokens transferred matches the quantity it reserved. The repair was already on SGLang most important in #36988, merged September 6 for a associated failure path however not but launched. The maintainers additionally merged #38935, which rejects these requests earlier, following our strategy.

We validated each modifications towards the researcher’s replica script and our personal canary-marker checks earlier than restoring the mannequin.

Background: how we serve GLM 5.3

Workers AI runs GLM 5.3 with SGLang in a disaggregated configuration: a pool of prefill GPUs processes the immediate and produces the mannequin’s KV cache for it, then arms that cache over distant direct reminiscence entry (RDMA), through Mooncake, to a pool of decode GPUs that generate the response. We have written about this structure earlier than in our posts on high-performance LLMs and smaller, faster, safer models.

The KV cache in GPU reminiscence is a shared pool of fixed-size pages. Each request is assigned pages when it’s admitted and releases them when it finishes. Pages aren’t at present zeroed between requests; the subsequent request that will get a web page merely overwrites it. This strategy is frequent amongst inference engines and is important to attain one of the best efficiency, but it surely signifies that the inference engine is accountable for monitoring possession of those reminiscence pages. KV cache web page possession and lifelong monitoring has to cope with not simply shared possession inside a node (KV cache offload and token era each entry the reminiscence with totally different lifetimes), but additionally possession that’s technically shared with the distant prefill node (which holds a simultaneous mutable reference to the web page).

Furthermore, the dimensions of this pool just isn’t a configuration worth. SGLang sizes it at startup from no matter GPU reminiscence stays after loading the mannequin weights. The mannequin’s marketed context size is a separate configuration worth. Nothing forces the 2 to agree, and it’s our job to examine that they do.

Issue one: misconfiguration

On our GLM 5.3 prefill nodes with our configuration, the pool got here to 681,280 tokens whereas the mannequin’s context size is 1,048,576 tokens. The chosen KV pool dimension is printed by SGLang in logs throughout startup and uncovered as a metric. We had been conscious of this dimension mismatch; nonetheless, throughout pre-launch testing we noticed profitable HTTP standing codes and a streaming physique with no error frames throughout benchmarking. We assumed that the engine was able to gracefully dealing with the distinction between the KV cache pool and requested context window through streaming or another mechanism.

This was one crucial hole in our testing: we solely validated HTTP response codes and engine efficiency at lengthy contexts, with out validating mannequin output and coherence. We now examine KV pool dimension towards context size straight, from the operating engine, earlier than any disaggregated deployment change ships, fairly than inferring it from response codes.

Issue two: SGLang bug

In v0.5.21, the SGLang group has now addressed this bug in the way it was dealing with prompts longer than the KV pool dimension. Here’s what was taking place earlier than.

When a immediate arrived that was longer than the KV pool might maintain, SGLang’s scheduler rejected it. The rejection path (Req.set_finish_with_abort) recorded the error to be returned to the shopper and swapped the immediate for a one-token placeholder in order that the request might drain by way of the conventional batch equipment with out allocating actual reminiscence. On a single-node deployment this labored: the placeholder ran, the batch completed, and the shopper acquired an error.

In disaggregated mode reminiscent of we function, the admission queues on each prefill and decode checked whether or not a request was completed, not whether or not it was rejected. A rejected request reported itself as reside. So:

1. Bootstrap. The router instructed the decode node the immediate size (8 pages within the diagram). Decode reserved 8 KV pages for the request.
2. Prefill admission. The prefill admission queue admitted any request the place req.completed() was False. A rejected request saved its rejection in to_finish, not finished_reason, so completed() nonetheless returned False. The rejected request was admitted.
3. Prefill ahead move. Rejection had already changed the immediate with a one-token placeholder. Prefill ran a trivial ahead move on that one token.
4. KV switch. Prefill despatched one web page of KV to decode and signaled that the switch was full.
5. Decode admission. Decode’s admission queue had the identical completed() examine, so it accepted the handoff.
6. Decode consideration. Decode began producing and a focus learn all 8 reserved pages. Only web page P0 had been written by this request. Pages P1 to P7 nonetheless held what earlier requests left in them: different customers’ prompts and partial generations.
7. Response. The shopper acquired HTTP 200 as a substitute of HTTP 400. prompt_tokens matched the lengthy immediate and the reply was coherent. It arrived quicker than an 8-page prefill ought to take, and it contained content material from different customers’ KV pages.

Every request over the edge behaved this manner. This bug had been reported to SGLang by one other consumer as concern #30233 on July 6, 2026, which Cloudflare observed whereas investigating this incident. We had been flagging these kinds of points with SGLang since v0.5.10 and this safety incident sped up work that Cloudflare had already began in collaboration with the SGLang group. The patch for this concern was launched in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22, and is accessible for all different operators depending on SGLang to make use of. We extremely suggest all operators replace SGLang dependencies to v0.5.22 or later. For extra particulars on particular PRs, learn on under.

GLM 5.3 was the one mannequin affected by this bug. Every different disaggregated deployment on Workers AI has KV swimming pools bigger than its context size on each prefill and decode, so no request might attain the rejection path. We have utilized the engine repair and the hardening described under to each mannequin we serve with SGLang, and our deployment checks require them for each new deployment.

Timeline

All occasions UTC.

Ongoing hardening effort

This bug belongs to a category of KV cache memory-safety bugs that Cloudflare has been working to handle for a number of months. In every case, a web page is freed, reused, or learn whereas one thing else nonetheless has a declare on it. We see two frequent causes.

The first is web page lifetime inside a single node. Several duties on a decode node can maintain the identical web page directly: token era, offload to host reminiscence, the KV switch receiver, and the abort and error paths. If one among them frees the web page whereas one other remains to be utilizing it, a brand new request could be on condition that web page whereas it’s nonetheless being learn or written. Our earlier fixes for decode offload (#24226, #30899) and double launch on grammar errors (#30937) had been of this type.

The second is a race throughout nodes. When a request is aborted whereas prefill remains to be writing its KV cache to decode over RDMA, the writes already despatched can’t be recalled. If decode frees these pages straight away, these pages could be given to a different request, and the late writes land in that request’s KV cache. Cloudflare has contributed to the hassle to repair this race by way of proposed fixes (#31357, #38961) and a replica script (#32564). In each instances the result’s corrupted output, or one consumer’s knowledge in one other consumer’s response, with no error raised. The bug on this incident was totally different: a rejected request went by way of the handoff each time, fairly than by a timing accident. But it had the identical consequence: decode studying pages this request by no means wrote. None of those coated the failure on this incident during which prefill rejected the request, and decode by no means discovered of it.

Since the incident, we’ve got:

• Extended deferred launch to carry prefill’s supply pages as properly, and to cowl each failure kind, not simply aborts. We proposed it upstream as #38961, which was outdated by #40645, #41402, and #41404.
• Deployed bounded Mooncake switch batches (#32758, from the SGLang group), so no single switch name can outlast the discharge timeout.
• Changed our load balancer so a shopper disconnect now not aborts a request mid-transfer. It holds the prefill connection till prefill finishes and the decode connection till the primary token, then closes each.
• Added a examine to each disaggregated deployment change: KV pool dimension measured towards context size, deferred launch on each roles, switch batch bounds inside the discharge timeout, and all required patches utilized.

All of those modifications had been launched to our disaggregated manufacturing deployments.

Working with the SGLang group

We met with the SGLang maintainers at RadixArk on September 10, the day after the incident. We agreed that RadixArk would examine the race circumstances we had reported, upstream the integrity-checking thought, and construct replica checks for this class of bug.

The following have landed on SGLang most important to handle this incident and associated KV cache race circumstances, and all however one are included in v0.5.22. Operators operating disaggregated SGLang ought to replace to v0.5.22 or backport these patches as wanted. 

These follow-ups are nonetheless works in progress:

• #39729 Additional cleanup for --disaggregation-enable-kv-checksum. Merged, awaiting launch.
• Zero KV pages on launch, draft proposal: #46435
• Extend KV checksum validation from the prefill-to-decode handoff to each decode step (see Integrity by development, under)
• Broader KV Canary and KV Checksum mannequin and have help

What we suggest to different operators

If you run SGLang in disaggregated prefill/decode mode, we suggest the next:

1. Measure your KV pool and examine it to your context size, on each prefill and decode node, from the engine’s startup log, after each change to weights, quantization, parallelism, or reminiscence fraction. If the pool is smaller, decrease the context size to suit. Do not depend on a proxy to implement the restrict, and don’t assume the engine will reject what doesn’t match. This is the examine we didn’t implement.
2. Upgrade to a construct that features #36988 and #38935 (v0.5.22 or later), so an over-length request is rejected earlier than handoff no matter pool dimension. This is the engine-side repair, and it’s the backstop for the day your examine in the first step is fallacious.
3. Run with deferred KV launch on each roles (default after #41023), and certain your Mooncake switch batches so a single timed-out name can not go away an unbounded tail of writes.
4. Turn on corruption detection the place your mannequin helps it. --disaggregation-enable-kv-checksum and --kv-canary don’t but cowl each mannequin structure. Check your logs to substantiate requests are being verified. We are working with the SGLang group to increase each checks to extra architectures, together with GLM 5.3, GLM 5.3 Flash, and DeepSeek V4.
5. Ensure your proxy just isn’t producing pointless aborts. Configure your load balancer to permit requests to proceed as much as the primary token, in order that you don’t set off an abort that requires cross-node coordination.

Longer time period

Integrity by development

Configuration checks and lifecycle fixes cut back the variety of methods a web page could be misused. They don’t make misuse unattainable, and this incident is a reminder that the subsequent bug may have a form we’ve got not seen. The sturdy reply is for the engine to detect, at learn time, {that a} request is about to attend over a web page it doesn’t personal, and fail that one request fairly than return another person’s knowledge.

We constructed a prototype of this over the summer time and described it in August. Every bodily KV web page carries a tag that modifications when the web page is reallocated, every request information the tags it expects, and the mapping is checked earlier than decode reads the cache. In our checks it value below 1% in throughput and tail latency. We examined it on GLM 5.2 in mid-August, however disabled it shortly afterwards because of reliability points.

The prototype was additionally tied to particular consideration backends and mannequin architectures, so it couldn’t be a normal repair. We are working with the SGLang group to construct a manufacturing model into the engine that works throughout each mannequin, fairly than sustaining Cloudflare-specific patches. The per-transfer checksum in #39500 is a primary step: it validates the handoff from prefill to decode. Page-level tags go additional, as a result of they examine each learn, together with reads lengthy after the switch.

Integrity by mathematical impossibility

RDMA copies knowledge straight from one node’s reminiscence to a different’s. It’s important for the 2 machines to agree on the place the info goes, how giant it’s, and when it’s completed; any mismatch may cause the receiver to learn the fallacious knowledge. This bug is an instance of a consistency failure in a distributed system, during which the 2 machines had inconsistent views of the state of that system.

Elsewhere in Cloudflare, we’ve got put effort into proving the protection of distributed techniques utilizing formal verification strategies. Topaz permits conflict-free IP choice for our authoritative DNS servers. Models of Meerkat’s QuePaxa implementation have demonstrated consistency bugs that had been subsequently fastened.

We are actually directing our expertise in formal strategies to KV cache switch engines. This will begin with modeling and proving the protection of an RDMA-based KV cache switch protocol. The objective is to mannequin our manufacturing SGLang’s distributed KV cache system utilizing Mooncake, then monitor and repair any divergence from the protection properties we wish to assure. The purpose is to show — so far as attainable — {that a} bug like this can not happen once more. Realistically, we count on gaps in what the provers can inform us, however proving the remainder of the system permits us to extra intently evaluation and monitor these gaps.

Coordinated disclosure

We are publishing this report as a part of a coordinated disclosure. We wish to thank the researcher, Gadi Evron, who reported this by way of our HackerOne program with a transparent, reproducible script and gave us time to repair it and to coordinate with the SGLang maintainers earlier than publishing. We additionally wish to thank the SGLang group at RadixArk, who handled this as their drawback in addition to ours and landed the upstream fixes with urgency.

Read extra at the post from the SGLang team.

If you discover a safety concern in Workers AI, please report it at hackerone.com/cloudflare.