KV-Cache Quantization for Long-Context LLM Inference in Production: INT8, FP8, and the Tradeoffs in vLLM

Long-context inference is often limited less by model weights than by the key-value cache, or KV cache. Every token generated during a request adds key and value tensors that must remain available for future attention operations. At a few thousand tokens, the memory cost may be manageable. At 32K, 128K, or more tokens—and when serving many concurrent users—the KV cache can consume most of a GPU’s usable memory. KV-cache quantization addresses this bottleneck by storing those tensors in lower-precision formats such as INT8 or FP8. In the right workload, it can approximately halve KV-cache memory usage, increase concurrency, and make longer context windows practical without changing the model’s main weights.

Why the KV cache becomes the production bottleneck

Transformer inference has two important phases. During prefill, the server processes the prompt and builds keys and values for every layer and token. During decode, it generates new tokens one at a time and repeatedly attends to the cached history. The cache prevents the server from recomputing the entire prompt for every generated token, but it also creates a growing memory allocation for each active sequence.

A simplified estimate for a standard decoder-only model is:

KV memory = 2 × layers × tokens × key/value heads × head dimension × bytes per element

The factor of two represents keys and values. The actual implementation also has block-management overhead, padding, alignment, and differences caused by multi-head, grouped-query, or multi-query attention. Grouped-query attention reduces the number of KV heads and therefore reduces cache size, but long sequences can still consume many gigabytes.

For example, a model with 32 layers, 8 KV heads, a head dimension of 128, and a 128K-token sequence requires roughly 8 GB for an FP16 cache before allocator and batching overhead. Several simultaneous requests can exceed the memory available after model weights, CUDA graphs, temporary activations, and workspace allocations are accounted for. When that happens, the server may evict requests, reduce concurrency, reject long prompts, or fall back to CPU swapping.

What KV-cache quantization changes

Quantization stores the cached key and value elements with fewer bits. FP16 and BF16 use 16 bits per element. INT8 uses 8 bits, and FP8 also uses 8 bits. In theory, either format cuts the raw cache footprint by about 50 percent. That does not mean total GPU memory usage is cut in half: model weights, activations, metadata, temporary buffers, and runtime overhead are unchanged.

The serving engine must quantize new cache entries as they are produced and dequantize them, or otherwise consume them through quantized attention kernels. This makes KV quantization different from weight-only quantization. Weight quantization is mostly a property of the model files and matrix multiplications. KV quantization is a live-memory and attention-kernel feature that affects every active request.

In vLLM, the principal configuration is typically exposed through the --kv-cache-dtype option. Depending on the vLLM version, hardware, and model, available choices can include automatic selection, FP8 variants, and INT8 modes. Similar controls exist in other serving stacks, although names and supported calibration methods differ. Always verify the installed engine’s documentation rather than copying a command from an older deployment guide.

FP8 versus INT8

FP8

FP8 is attractive on newer accelerators because supported GPUs can execute FP8 operations using native hardware paths. The format preserves an exponent, giving it more dynamic range than a simple integer representation. That matters because key and value distributions can vary across layers, heads, positions, and prompts.

On compatible NVIDIA Hopper and Ada-generation systems, as well as supported accelerators from other vendors, FP8 can provide a useful combination of lower memory use and competitive throughput. vLLM commonly exposes FP8 KV-cache modes through values such as fp8, with the exact variant depending on the backend. Some formats use different exponent and mantissa layouts, so “FP8” is not one universal numerical behavior.

FP8 can be used with runtime scaling, but calibrated scales may produce more consistent accuracy. A calibration process measures representative activations and selects scaling factors that reduce clipping and under-utilization of the available range. Calibration data should resemble production prompts, especially if the application relies on long documents, code, multilingual text, or structured retrieval.

INT8

INT8 has a longer history and can be useful on hardware where FP8 support is limited or unavailable. The cache values are represented as integers plus scale information. The critical implementation detail is how those scales are chosen.

Per-tensor scaling uses a shared scale for a larger block of values. It has low metadata overhead and can be efficient, but outliers in one portion of the tensor may force a scale that wastes precision elsewhere. Per-token or finer-grained scaling gives each token, row, or block a more suitable range, usually improving accuracy at the cost of extra scale metadata and additional computation.

INT8 is not automatically better on older hardware. If the GPU lacks efficient low-precision cache kernels, quantization and dequantization can add latency or consume bandwidth savings. An INT8 configuration that looks efficient in a memory calculation may be slower than FP16 in a real benchmark because attention kernels, scale loads, and format conversions dominate.

Accuracy risks are concentrated in long-context behavior

The main risk is not usually an immediate failure on short prompts. Quantized KV caches can pass ordinary functional tests while losing accuracy on the workloads that motivated the change: long-context retrieval, multi-document question answering, code navigation, summarization of lengthy inputs, and conversations with important facts far back in the history.

Quantization error accumulates through repeated attention operations. A small perturbation in cached keys or values may be harmless for one layer and one token, but the effect can become measurable across many layers and thousands of decode steps. Outliers are particularly important. If a small number of values have unusually large magnitude, a coarse scale can either clip them or reduce the effective precision available to the majority of values.

Evaluate more than perplexity. Run retrieval tests with known answer locations, long-context needle tests, exact-match checks for structured output, tool-call validation, code-generation tests, and representative conversational sessions. Compare both answer quality and refusal or hallucination rates. Test different prompt lengths because a configuration that is indistinguishable at 8K tokens may degrade at 64K.

Performance tradeoffs in vLLM and similar stacks

Memory savings can improve throughput indirectly by allowing more sequences in continuous batching. If the deployment is memory-bound, this is often the biggest benefit. The scheduler can keep more requests resident, avoid swapping, and use a larger token budget per batch.

However, lower memory use does not guarantee lower latency. Quantization introduces scale handling, packing or conversion, and potentially different kernel selection. Prefill and decode may behave differently. Decode is often memory-bandwidth-sensitive, so smaller cache elements can help. Prefill is more compute-intensive and may see little benefit, or even a regression, depending on the attention implementation and head dimension.

Speculative decoding deserves separate testing. The draft model and verification model may have different cache formats, and quantization error can affect token acceptance rates. If acceptance falls substantially, the theoretical benefit of speculative decoding can disappear. Prefix caching also needs validation: cached prefixes must use compatible dtype and scale metadata, and cache reuse should not silently increase conversion work.

Measure time to first token, inter-token latency, throughput, GPU memory, active sequences, preemption rate, and tokens per second at several concurrency levels. A single benchmark with one short prompt is not sufficient. Include cold starts, mixed prompt lengths, streaming responses, and the maximum context length your product will actually accept.

Operational recommendations

Start with a memory model. Estimate cache usage from layers, KV heads, head dimension, token count, and concurrency before changing precision.
Use native FP8 when the GPU and kernels support it. Confirm that the serving stack is using an optimized attention path rather than emulating the format.
Prefer finer-grained scaling when accuracy requires it. Compare per-token, per-channel, per-block, and per-tensor options supported by the engine.
Calibrate with production-like data. Include the longest prompts and the domains where errors are expensive.
Keep an FP16 or BF16 rollback path. Make the KV-cache dtype a deployment configuration, not a permanent model conversion.
Set a realistic context limit. Quantization increases capacity, but it does not eliminate attention cost or guarantee useful model behavior at the maximum window.
Watch memory fragmentation and scheduler behavior. A theoretical 50 percent cache reduction may produce a smaller practical gain if blocks, scales, and temporary buffers dominate.

How to decide whether it is worth deploying

KV-cache quantization is most valuable when GPU memory limits concurrency or prevents the application from accepting required context lengths. If the workload uses short prompts, low concurrency, or a GPU with substantial unused memory, the complexity may not justify the change.

For long-context production systems, a sensible rollout is to benchmark BF16 or FP16 first, then test FP8 on supported hardware, followed by INT8 alternatives where they offer a compatibility or accuracy advantage. Compare quality at the actual context lengths, not merely on short samples. If accuracy remains within the product’s tolerance and latency is stable, the memory headroom can be converted into longer contexts, more concurrent users, fewer preemptions, or a smaller GPU footprint.

The practical conclusion for September 2026 is straightforward: treat KV-cache quantization as a serving-system optimization, not a free model upgrade. INT8 and FP8 can make long-context inference substantially more economical, but the result depends on scaling strategy, attention kernels, GPU generation, scheduler behavior, and workload-specific accuracy. Production success comes from measuring the complete serving path and preserving a fast, observable rollback when the cache dtype does not behave as expected.

Search This Blog

Technology On the Net