Prefill/Decode Disaggregation for Agentic LLM Serving in September 2026

Agentic applications put unusual pressure on LLM infrastructure. A single user request may trigger planning, tool calls, retrieval, code execution, verification, and several follow-up model calls. Some requests contain very long prompts but produce only a few tokens; others have modest prompts but generate long, tool-oriented responses. Running every request through one undifferentiated inference pool makes it difficult to control both time to first token (TTFT) and inter-token latency (ITL). Prefill/decode disaggregation addresses that mismatch by separating prompt processing from token generation, allowing each stage to be scaled and tuned independently.

Image: Al Piskun via Wikimedia Commons (CC BY-SA 4.0)

Image: The Original Benny C via Wikimedia Commons (CC BY-SA 4.0)

What prefill/decode disaggregation actually separates

During prefill, the model reads the prompt and computes attention states for all input tokens. Those states become the key-value (KV) cache used during generation. Prefill is generally compute-heavy and tends to benefit from high-throughput GPUs and batching. Long agent traces, retrieved documents, tool results, and accumulated conversation history can make this phase especially expensive.

During decode, the model generates one new token at a time. Each step reads the existing KV cache and appends new key-value entries. Decode is often constrained by memory bandwidth, scheduling overhead, and the number of concurrent sequences. Its user-visible metric is usually ITL: how quickly tokens continue arriving after the first token.

In a monolithic deployment, the same GPU pool handles both phases. A large prompt can consume compute and memory bandwidth while short responses are waiting to decode. Conversely, a busy decode workload can prevent long prompts from receiving efficient batching. Disaggregation creates separate prefill and decode pools. A request is prefetched on one worker, its KV cache is handed to a decoder, and generation continues there.

When splitting the stages is worth it

Disaggregation is not automatically faster. It introduces a network transfer, coordination protocol, more services, and more failure modes. It is most useful when the workload has a persistent imbalance between prompt processing and generation.

• Long or variable prompts: Retrieval-augmented generation, code agents, browser agents, and long conversations often create large prefill bursts.
• Strict TTFT targets: A dedicated prefill pool can absorb prompt-heavy traffic without allowing decode work to dominate scheduling.
• Strict streaming targets: Dedicated decode capacity makes ITL more predictable for interactive responses.
• Different hardware requirements: Prefill and decode may have different optimal GPU counts, memory configurations, or parallelism settings.
• Agentic fan-out: One user turn may create several model calls with different prompt lengths and output budgets.
• Capacity isolation: Batch, offline, and interactive traffic can be assigned to different pools rather than competing in one scheduler.

A simple first test is to measure the workload before changing the architecture. Track prompt-token distribution, generated-token distribution, TTFT, ITL, queueing time, KV-transfer time, GPU utilization, and cache hit rate. If prefill and decode consume similar resources and the service is not queueing, a split may add complexity without improving the user experience.

How the KV handoff works

The prefill worker receives the original prompt and runs the model through the input tokens. Instead of discarding the resulting KV cache, it registers the cache blocks with a transfer connector. The orchestration layer then routes the request to a decode worker and supplies enough metadata for that worker to locate and import the corresponding blocks.

The transfer can use GPU-to-GPU paths, shared memory, CUDA IPC, TCP, or RDMA-backed transports, depending on the deployment. The practical goal is to move KV data without converting it into ordinary application-level text or repeatedly recomputing the prompt. The decoder must know the request identity, model and tokenizer compatibility, sequence length, block layout, tensor or pipeline parallelism assumptions, and the location or handle for each cache segment.

NIXL, the NVIDIA Inference Xfer Library, is commonly used as a high-performance transport in this design. LMCache can use NIXL for prefill/decode transfer while also providing broader KV-cache management, including reuse and offload patterns. The exact connector names and configuration fields depend on the installed vLLM and LMCache versions, so production teams should pin compatible versions and validate the configuration against the release documentation rather than copying a command from an older deployment.

Correctness matters as much as bandwidth. A KV cache generated with a different model revision, tokenizer, quantization scheme, RoPE configuration, or parallelism layout may be unusable. Cache ownership also has to be explicit: the prefill side is normally the producer or sender, and the decode side is the consumer or receiver. A request should not be admitted to decoding until the required cache blocks are available or the system has a defined fallback path.

vLLM AgentX and agent-oriented serving

Agent workloads make the split more valuable because the serving system must deal with more than a single completion. An agent runtime may maintain a stateful trace, issue tool calls, resume a paused generation, and start follow-up requests that share substantial context. In this setting, the serving layer needs request-aware routing rather than a simple round-robin load balancer.

vLLM AgentX-style deployments should be treated as an orchestration problem around the inference engine. The agent layer owns task state, tool execution, cancellation, deadlines, and retries. The prefill/decode layer owns model execution and KV movement. Keep those responsibilities separate. Do not make a tool executor responsible for guessing whether a KV cache is still valid, and do not let a proxy retry a generation after the agent runtime has already committed an external side effect.

For agent traces, record a stable logical request identifier and a separate attempt identifier. A retry may need to recompute prefill if the original cache was lost, while a resumed generation may reuse a valid cache. Observability should distinguish these cases so that cache misses are not mistaken for model latency.

LMCache NIXL P/D deployments

LMCache’s prefill/decode mode typically uses a sender on the prefiller and a receiver on the decoder, with NIXL acting as the transfer path. A deployment normally includes a prefill service, a decode service, and a proxy or coordinator that connects the two. Depending on the configuration, LMCache can also combine remote transfer with local reuse or offload through a multi-connector setup.

Start with the smallest topology that proves the handoff: one prefill worker, one decode worker, and a controlled test client. Confirm that the decoder receives the expected token count, that generated output matches a non-disaggregated baseline, and that transfer time is visible as its own metric. Only then add multiple workers, autoscaling, RDMA, or cross-node placement.

Ray Serve LLM as the deployment layer

Ray Serve LLM provides a higher-level way to package prefill and decode replicas. Its P/D abstractions can expose a single OpenAI-compatible application while deploying separate server roles behind the scenes. This is useful when teams want declarative replica management, placement groups, autoscaling, and integration with a larger Ray application.

The important operational detail is that Ray does not remove the underlying systems constraints. The prefill and decode configurations still need compatible vLLM settings, connector configuration, model artifacts, and network access. Autoscaling should be based on stage-specific signals. Prefill replicas may need to scale on prompt queue depth or input-token rate, while decode replicas should respond to active sequences, output-token rate, or ITL. Scaling both pools from one generic request count can create the wrong capacity.

Production deployment checklist

• Pin compatible versions of vLLM, LMCache, NIXL, CUDA, drivers, and container images.
• Verify identical model weights, tokenizer files, quantization settings, and parallelism assumptions.
• Define producer, consumer, and coordinator roles explicitly.
• Measure KV-transfer latency separately from prefill and decode latency.
• Test the actual network path, including cross-node bandwidth, RDMA availability, GPU peer access, and fallback behavior.
• Set bounded queues, request deadlines, cancellation propagation, and overload responses.
• Design for cache loss: a decoder should be able to trigger recomputation or return a controlled retryable error.
• Use request and attempt identifiers to make retries and resumed agent steps traceable.
• Protect the transfer channel with network policy, authentication, and tenant isolation where applicable.
• Load-test realistic agent traces, not only isolated short prompts.
• Compare cost per completed request, not just tokens per second.
• Roll out gradually with a feature flag and retain a monolithic fallback.

The practical decision

Split prefill from decode when prompt processing and generation have meaningfully different capacity needs, when TTFT and ITL must be controlled independently, or when agent traces create bursty, long-context traffic. Use vLLM connectors and LMCache/NIXL when direct KV movement is the bottleneck, and use Ray Serve LLM when you need a managed deployment model around separate inference roles.

The architecture succeeds only when the handoff is treated as a first-class production protocol. Cache compatibility, transfer latency, retries, cancellation, observability, and security are not implementation details to postpone. For many agent systems, the best rollout is incremental: establish reliable KV transfer with fixed replicas, validate quality and latency against a monolithic baseline, then add stage-specific autoscaling and cache reuse once the operational behavior is understood.