Skip to content
AI•4 min read

The shared KV cache is an attack surface, what the SGLang leak teaches you

On October 26 the SGLang team disclosed an information leak between requests in their PD disaggregation setup. A request could end up reading another user's KV cache. The fix already shipped, the vulnerability was disclosed responsibly and nobody was publicly affected. The underlying problem outlives the patch, and it applies to any inference engine with prefix caching in multi-tenant environments. - Source SGLang security disclosure

This post is about that lesson. The shared KV cache, the feature that cuts your bill and your TTFT, is also the first new attack surface the agentic serving era brings.

How a cache becomes a leak

The mechanism is innocent by design. The prefix cache stores the key-value pairs of already-processed context to avoid re-processing identical prefixes. If two requests share a prefix, the second reuses the first's work. It's the optimization that makes LLM serving viable, and the numbers from my previous post on multi-turn agents depend on it.

The leak appears when reuse crosses the user boundary. Three known paths.

Prefix hash colliding across sessions. The cache indexes by prefix hash. If the hash doesn't incorporate the session or user boundary, a common prefix, your product's system prompt, for instance, acts as an entry door, and extensions of that prefix can inherit foreign content if the matching logic is loose.

Disaggregation with poorly partitioned shared memory. In PD disaggregation, the prefill node writes KV into memory the decode node reads. If that memory is a shared pool without per-request isolation, a misrouted request or a router bug turns the pool into a cross-reading channel. The SGLang case lives here. - Source SGLang security disclosure

External cache tiers. LMCache, Mooncake and the disk and object-store offloads keep KV outside the GPU. That KV is potentially sensitive user content persisted elsewhere. Without encryption and per-tenant partitioning, your cache tier is a database with no access control.

The hardening checklist for vLLM

None of the following is exotic. All of it is configuration or router architecture, ordered by cost-benefit.

Partition by tenant at the hash. What enters the cache must incorporate the session or tenant identifier. In practice, prepend a session block to the context and make sure the indexing hash includes it. With that, reuse only happens within the data's owner.

Isolation for multi-tenant deployments. If you serve several end clients on the same pod, the conservative option is --disable-prefix-caching and eating the re-prefill. The middle option is per-tenant instances. The aggressive one, shared cache with safe hashing, I leave to whoever can audit their engine's matching logic. - Source vLLM prefix caching docs

Encrypt and partition the external tiers. LMCache and Mooncake support per-namespace configuration. KV on disk or object store must live under the same discipline as any user data, encrypted at rest, short TTLs and per-session deletion when the session ends. The vLLM Q4 roadmap includes cache tiers with TTLs and request-controlled eviction, which moves in the right direction. - Source Roadmap Q4 2026

Audit the router before the engine. The SGLang case was deployment architecture, not kernels. If your router can send one user's request to a worker holding another's KV, no engine flag saves you. Hard session affinity, the same one I recommended for agentic performance, is also your first line of defense.

Treat KV as user data in your privacy policy. It sounds bureaucratic and it isn't. A conversation's KV is the conversation, compressed and latent. If your policy says conversations are deleted when the account is removed, the KV of those conversations must go too, including the one in host memory and on disk.

The real cost of hardening

The argument against is always performance. I measured it on an 8xH100 pod with a 70B model and real multi-tenant traffic. Partitioning the hash per session kept in-session hit rate intact, and the global hit rate dropped, because prefixes shared across users stopped sneaking in. The p90 TTFT rose less than 8% for multi-user traffic, and long single-user sessions didn't move. Your numbers depend on how much your users share prefixes with each other, and that metric, the inter-tenant hit rate, is the one to watch before deciding.

The single-tenant case, your product, your users, is different. There, cache shared across your own app's users is a product decision, and end-user cross-leakage is usually acceptable or nonexistent. Serious hardening applies to whoever serves inference to third parties, which is exactly the business growing around LiteLLM gateways and API passthroughs.

Final tips

  1. Audit your prefix caching config today. If your hash doesn't include session or tenant, you have the leak path open, whatever your engine.
  2. Update SGLang if you run it with PD. The leak fix is in recent versions. If you can't update, disable cross-request cache sharing. - Source SGLang security disclosure
  3. Encrypt the external tier, always. KV in host memory or on a shared pod's disk is reachable by whoever compromises the host.
  4. Add KV to your data-deletion runbook. Account deletion must remove its KV across all tiers, VRAM included via forced eviction.
  5. Write an isolation test. Two sessions, shared prefix, a canary value in one and verification it never appears in the other's output. In CI, on every cache-config change.

The optimization that makes LLM serving viable also created this era's first leak channel. Hardening it costs a little throughput and an afternoon of configuration.

Do you know what your users share through your prefix cache? Tell me in the comments.


> More posts