Tiered KV Cache Offloading for Production LLM Serving in September 2026: When to Spill from GPU HBM to Host Memory and Storage

For production LLM serving, GPU memory is no longer reserved only for model weights. The key-value cache, or KV cache, can consume more memory than the model itself when users send long prompts, maintain large conversations, or request many generated tokens concurrently. Tiered KV cache offloading provides a way to keep serving when GPU HBM is full by moving less-recently-used cache blocks to host RAM and, when necessary, to fast local or remote storage. The trade-off is straightforward: every tier increases available capacity but adds latency, bandwidth pressure, and operational complexity.

Why the KV cache becomes the production bottleneck

During transformer inference, the model computes attention over tokens that have already been processed. Instead of recomputing those tokens for every generation step, the server stores intermediate key and value tensors in the KV cache. Each active request therefore consumes cache memory as its prompt and response grow.

The memory requirement depends on the number of layers, attention heads, head dimension, data type, and number of tokens. A simplified estimate is:

KV cache memory ≈ tokens × layers × 2 × key-value elements per token × bytes per element

The factor of two represents keys and values. The exact calculation varies with grouped-query attention, multi-query attention, quantization, and the engine's memory layout, but the operational result is consistent: a long-context workload with high concurrency can exhaust HBM even when the model weights fit comfortably.

GPU memory pressure creates several failure modes. A scheduler may stop admitting requests, evict cached prompt prefixes, reduce batch size, or report an out-of-memory error. Some systems appear healthy during short tests because their prompts are small, then become unstable when real traffic includes long documents, tool traces, or multi-turn sessions.

What “tiered” KV caching means

A tiered cache treats memory as a hierarchy rather than a single pool. The fastest tier is GPU HBM, where attention kernels can access cache blocks with the lowest latency. The next tier is host memory, normally system DRAM connected to the GPU over PCIe or a coherent interconnect. A slower tier can use local NVMe, a memory-mapped file, a dedicated cache service, or another storage system.

• GPU HBM: Best for active sequences and frequently reused prefixes. It has very high bandwidth and predictable access latency, but limited capacity.
• Host DRAM: Useful for cold or temporarily inactive blocks. It offers much more capacity than HBM, but transfers compete with model inputs, outputs, and other device traffic.
• Local NVMe: A capacity safety valve for large prompt caches and low-frequency reuse. It is not a replacement for HBM during every decode step.
• Remote storage or cache services: Appropriate for sharing reusable data across workers or nodes, but network latency and serialization make them unsuitable for latency-sensitive cache misses unless access is carefully controlled.

The important design principle is that offloaded KV blocks should not be treated as if they were still resident on the GPU. A request whose next attention operation needs a block in host memory must wait for that block to be transferred back, unless the engine supports an execution strategy that overlaps data movement with computation.

When host-memory offloading is a good idea

Host-memory offloading works best when GPU cache pressure is bursty rather than constant. For example, a service may normally fit its active requests in HBM but experience brief spikes when several long-context requests arrive together. Moving inactive blocks to DRAM can preserve admission capacity without permanently shrinking the GPU batch.

It is also useful for workloads with a large working set and a smaller hot set. A chat application may maintain thousands of sessions, but only a fraction generate tokens at any moment. Keeping recently active sessions in HBM and moving idle sessions to host memory can improve utilization while avoiding repeated prompt processing.

Prefix reuse is another strong use case. If many requests share a system prompt, policy document, or retrieval template, the cache can retain those blocks outside HBM and promote them when a matching request arrives. This is valuable only when the reuse rate is high enough to justify the transfer. If a prefix is read once and never used again, storing it in a lower tier merely adds work.

When host memory is not enough

Host DRAM should not be used as an unlimited extension of HBM. PCIe bandwidth is finite, and decode workloads repeatedly access KV data. If every token requires large transfers from host memory, the GPU may spend more time waiting for data than performing matrix operations. Tail latency rises first, followed by lower throughput and queue buildup.

Storage offloading is even more sensitive. NVMe can provide substantial capacity, but its latency is measured in microseconds or milliseconds rather than GPU memory cycles. It is generally appropriate for inactive sessions, long-lived prefix caches, or recovery-oriented capacity—not for blocks required on every decode iteration.

A practical policy is to keep the active decode window in HBM, use DRAM for warm sequences and reusable prefixes, and send only cold data to storage. The engine should know which blocks are likely to be requested soon rather than evicting solely by age.

How this applies to vLLM and similar engines

vLLM's paged attention design divides the KV cache into blocks instead of requiring each request to occupy one contiguous memory region. This makes allocation and reuse more flexible and provides a natural unit for eviction or transfer. Related engines use comparable block-based approaches, even when their scheduler and cache APIs differ.

When evaluating a vLLM deployment, distinguish among three separate mechanisms:

1. GPU KV-cache sizing: The fraction of HBM reserved for cache after accounting for model weights, CUDA graphs, activations, temporary buffers, and runtime overhead.
2. CPU offloading: Moving model weights or cache data to host memory. These are different mechanisms and have different performance effects.
3. Prefix caching: Reusing KV blocks for identical prompt prefixes. Prefix caching can reduce computation, while offloading determines where reusable blocks reside.

Configuration names and support levels change between releases, so operators should verify the installed vLLM version and inspect its current documentation before copying flags from an older deployment. The same caution applies to SGLang, TensorRT-LLM, llama.cpp-based servers, and managed inference platforms. A setting that controls CPU swap space may not have the same semantics as true asynchronous KV transfer.

Choosing an offload threshold

Do not wait until HBM reaches 100 percent before starting eviction. A production server needs headroom for new requests, temporary allocations, CUDA graph behavior, and scheduling variation. A common starting point is to reserve a safety margin and begin moving cold blocks when the GPU cache reaches a high-water mark, such as 80 to 90 percent of its configured capacity.

The correct threshold depends on the workload. Low-latency interactive serving needs more headroom because a single cache miss can affect time to first token or inter-token latency. Batch jobs can tolerate more aggressive eviction because throughput matters more than individual request latency.

Use separate policies for prompt processing and decoding when possible. Prompt processing can tolerate larger transfers because it already performs substantial computation, while decode is frequently memory-bound and sensitive to every stall. A cache block needed repeatedly during decode should be promoted and pinned until the request leaves the active batch.

Measure the right production signals

GPU utilization alone is not enough. A server can show high utilization while requests experience poor latency because transfers and kernel execution are contending for bandwidth. Monitor:

• HBM allocation, free headroom, and cache block occupancy
• Host-memory cache usage and eviction rates
• Storage read and write bandwidth, queue depth, and latency
• Cache hit rate by tier
• Bytes transferred per request and per generated token
• Time to first token and time between output tokens
• Queue wait time, preemption count, and request rejection rate
• Tokens per second at both request and system level
• p50, p95, and p99 latency, not just averages

Test with production-shaped traffic. Include long prompts, repeated prefixes, abandoned requests, simultaneous arrivals, and multi-turn conversations. A synthetic benchmark with uniform short prompts will usually overestimate the benefit of offloading.

Operational safeguards

Set explicit limits for each tier. Without a host-memory limit, the cache can compete with the operating system, data loaders, monitoring agents, and other services. Without a storage limit, a cache may fill the disk and affect logs or container runtime operations.

Use admission control when the lower tiers are saturated. It is better to reject or defer a request with a clear capacity signal than to allow uncontrolled swapping that damages every request's latency. Cache data should also be disposable: never treat KV blocks as durable application state, and never rely on them for correctness.

Finally, isolate tenants when necessary. Prefixes can contain sensitive user or business data, so cache keys must include the appropriate model, tokenizer, prompt configuration, tenant, and authorization context. A cache hit must never expose another customer's conversation merely because two prompts share a textual prefix.

The practical decision rule

Spill KV blocks from HBM to host memory when the blocks are temporarily inactive, likely to be reused, and the transfer can occur without disturbing the active decode batch. Spill to storage only when capacity is more important than immediate latency and the data is cold enough that a future miss is acceptable. If the workload constantly depends on offloaded blocks, the system is not benefiting from tiering—it is undersized for its working set.

In September 2026, tiered KV cache offloading is best viewed as a scheduler and capacity-management feature, not a magic way to turn a small GPU into a large one. The strongest deployments keep hot attention state close to the GPU, use host RAM for warm capacity, reserve storage for cold reuse, and validate every policy against tail latency and transfer bandwidth. The goal is not maximum cache capacity. The goal is predictable serving under the traffic patterns that actually reach production.