Skip to content
Kevin Davila mascotkevindavila.dev> SOFTWARE ENGINEERES
Angular•3 min read

KV cache that doesn't fit: offloading in vLLM, from file to object

Your multi-turn agents resend the full history every turn, and prefix caching saves you as long as sessions fit the GPU. The problem arrives when the number of conversations grows: HBM fills up, vLLM evicts old KV cache, and the next turn of that session pays a full prefill recompute. Offloading is the answer: store that evicted KV in CPU memory, disk, or other nodes, and reload it instead of recomputing. The tiered offloading framework has existed since vLLM v0.22 and shipped with an official guide this September. Source: vLLM blog, September 10, 2026.

The memory ladder

The design is host-centric: all KV flows through host RAM. When evicting from the GPU, the copy to CPU is a fast DMA and frees HBM immediately; from RAM, chunks cascade down to secondary tiers. On reload, RAM is checked first (it's a real LRU cache, not a staging buffer), then the tiers in order: filesystem, object storage, or remote peers over RDMA.

vllm serve Qwen/Qwen3.6-35B-A3B \
    --kv-transfer-config '{
        "kv_connector_extra_config": {
            "spec_name": "TieringOffloadingSpec",
            "cpu_bytes_to_use": 107374182400,
            "secondary_tiers": [{"type": "fs", "root_dir": "/mnt/kv-cache"}]
        }
    }'

That command reserves 100 GB of RAM for the primary tier and mounts a filesystem tier on NVMe. The object storage tier exists for S3/MinIO when you want cheap shared cache, and the P2P tier shares KV across instances over RDMA — that's what feeds the prefill/decode disaggregation in the next chapter.

The detail that turns this into infrastructure rather than a trick: chunks are named by content (hash of the token sequence). Two instances mounting the same storage share cache with no extra configuration, and any request whose prefix matches — same system prompt, same initial history — is a hit.

The numbers that justify the flag

The framework's official benchmark: Qwen3.6-35B-A3B on 2×H100, multi-turn conversations with 12K initial tokens plus 4K per turn, 8 turns, 64 concurrency. The scale of the problem by simultaneous conversations:

  • Up to ~64: HBM holds the working set. Every method performs fine.
  • 64 to 128: HBM fills and throughput craters without offloading. CPU offload holds the line.
  • Beyond 128: RAM fills too. The storage tier sustains a high hit rate and more than doubles throughput against the alternatives.

The math is simple: a disk hit takes milliseconds; a 12K-token prefill recompute takes seconds. Storage wins even though its latency is the worst in the system. Reproduction scripts: fs-offload-experiments.

Measure it like everything else

The framework exposes its metrics on the usual /metrics endpoint: primary tier utilization, transfer throughput, per-tier latencies and, the one that actually matters, per-tier hit rates. If your CPU hit rate hovers around zero, your workload doesn't reuse prefixes and offloading is just occupying your RAM. If the storage one climbs as conversations pass 128, the ladder is doing its job.

For orchestrators, the framework emits KV events: which chunks were stored or evicted, from which tier. Cache-aware routers like llm-d or Dynamo consume them to send each request to the instance most likely to hold its cache — cache-aware scheduling instead of blind round-robin.

Checkpoint and the typical error

Checkpoint: your server runs with TieringOffloadingSpec, you have an NVMe volume for the filesystem tier, and you watch per-tier hit rates in Prometheus. The test: double the conversation load and watch whether throughput craters (without offloading) or degrades gently (with it).

The typical error: sizing cpu_bytes_to_use by feel. That number competes with the rest of the system for RAM: give it all the host memory and the filesystem tier starts swapping, with reload latencies eating the gain. Start at 25% of RAM and go up while watching the primary tier utilization metric.

Next chapter

One GPU, two, four... and the real production question: should prefill and decode live on separate machines? The guide closes with multi-GPU and disaggregation.

> More posts