Chunked Prefill for Production LLM Serving in September 2026: How to Overlap Prefill and Decode Without Starving Interactive Traffic

Production LLM serving has an uncomfortable scheduling problem: the request that needs the most computation is often the request that users notice least if it runs slowly. A long prompt can consume a large batch of GPU time during prefill, while a short interactive request may already be waiting for its next decoded token. Chunked prefill addresses this mismatch by dividing prompt processing into smaller pieces and scheduling those pieces alongside decode work. Done correctly, it improves GPU utilization without allowing large-context requests to monopolize the system.

The scheduling problem: prefill is large, decode is sensitive

LLM inference has two operational phases. During prefill, the server processes the input prompt and builds the key-value cache used for generation. Prefill is highly parallel: thousands of input tokens can often be processed efficiently in a small number of large matrix operations. During decode, the server generates output one token at a time. Each decode step is comparatively small, but it is repeated for every active request.

These phases have different service-level objectives. Prefill determines time to first token, or TTFT. Decode determines inter-token latency, often called time per output token. A large prefill batch can produce excellent GPU throughput while temporarily delaying decode iterations. That delay is visible as a pause in an otherwise streaming response. In a chat application, even a few delayed decode steps can make the interface feel unresponsive.

Without careful scheduling, the server tends to favor one of two bad outcomes. It can run a large prefill operation to completion, maximizing arithmetic efficiency but starving active generations. Or it can prioritize decode so aggressively that long prompts enter the queue slowly and GPUs spend too much time on small, inefficient work. Chunked prefill is a compromise based on bounded, preemptible units of prompt computation.

What chunked prefill actually does

Instead of treating a prompt as one indivisible prefill job, the scheduler assigns it a token budget for the current iteration. For example, a 16,000-token prompt might be admitted in chunks of 512, 1,024, or 2,048 tokens. After one chunk is processed, the scheduler can run a decode step for existing requests before admitting the next chunk.

The key is not merely splitting the input. The serving engine must preserve the request's partially constructed KV cache, track how many prompt tokens have been processed, and resume the prefill at the correct position. The scheduler also needs to combine token work from different phases without exceeding the model's memory and execution limits.

A simplified scheduling loop looks like this:

Reserve capacity for decode tokens from active sequences.
Use remaining token and memory capacity for one or more prefill chunks.
Execute the mixed batch.
Update KV-cache state and request progress.
Repeat while respecting latency and fairness policies.

Some engines call this continuous batching with chunked prefill. The exact implementation varies, but the production goal is the same: keep decode work moving while allowing prompt processing to make steady progress.

Why naive chunk sizes fail

Chunk size controls the trade-off between throughput and responsiveness. Very large chunks make prefill efficient, but they create long scheduling quanta. A decode request that arrives just after a chunk starts may wait until the entire chunk completes. Very small chunks improve responsiveness but introduce more scheduling overhead, reduce kernel efficiency, and may prevent the GPU from reaching useful occupancy.

There is no universal best value. The right size depends on the model, GPU, sequence length distribution, quantization, batching strategy, and target latency. A 512-token chunk may be appropriate for a latency-sensitive assistant, while a batch-oriented summarization service may prefer a larger value.

Measure the effect using at least these metrics:

TTFT: time from request admission until the first generated token.
Inter-token latency: the distribution of time between streamed output tokens.
Decode p95 and p99: tail latency matters more than the average for interactive traffic.
Prefill throughput: prompt tokens processed per second.
GPU utilization: useful as a diagnostic, but not as a success metric by itself.
Queueing delay: separated by request class and phase.
KV-cache pressure: including evictions, recomputation, and allocation failures.

A configuration that raises average tokens per second while worsening decode p99 may be a regression for a customer-facing product.

Protect decode with a reservation

The most important production policy is to reserve scheduling capacity for active decode sequences. A scheduler should not fill every available token slot with prefill work and hope that decode fits afterward. It should estimate the decode demand for the current step, reserve that capacity, and admit prefill only into the remaining budget.

One practical model is a two-part budget:

A protected decode budget sized for the currently active sequences and their latency target.
A flexible prefill budget that consumes leftover compute and memory capacity.

The reservation may be expressed in tokens, estimated GPU time, or a combination of both. Token counts are simple, but they do not always represent equal work. Attention cost, sequence length, model architecture, speculative decoding, and hardware kernel behavior can make one token more expensive than another. Production systems should calibrate scheduler estimates against observed execution time rather than assuming a fixed cost per token.

Priority should also be separated from fairness. Interactive decode can receive strict latency protection, while prefill requests can be scheduled using weighted fair queuing, aging, or deadline-aware policies. Aging prevents a long prompt from remaining indefinitely at the bottom of the queue. Tenant-level quotas prevent one customer from submitting enough long-context work to consume every prefill slot.

Admission control is part of the design

Chunked prefill cannot solve an overloaded system by itself. If requests arrive faster than the GPU can process them, smaller chunks only distribute the delay. Admission control must enforce limits on concurrent sequences, prompt length, maximum output tokens, and total KV-cache allocation.

Use separate queues or virtual queues for interactive, batch, and background traffic. An interactive request should not compete directly with an offline evaluation job that has no meaningful TTFT requirement. The scheduler can then assign each class a policy, such as a maximum decode latency, a minimum service share, or a maximum amount of prefill work per round.

Rejecting or delaying work is preferable to accepting it and allowing every request to miss its latency objective. Return a clear overload response, use bounded retry behavior, and apply backpressure at the gateway. Otherwise, retries can multiply the load and create a positive feedback loop.

KV-cache memory changes the scheduling equation

Prefill chunks consume compute, but they also grow the request's KV cache. A scheduler that considers only GPU utilization can admit too many long prompts and fail later when decode sequences need additional cache space. Memory planning must account for both existing decode requests and prompts that are still being prefetched.

Use paged or block-based KV-cache allocation when supported by the serving engine. It reduces fragmentation and makes it easier to allocate cache incrementally as chunks complete. Track reserved, allocated, and reclaimable cache separately. A request that has finished prefill but is generating a long answer is still a memory resident workload.

When memory pressure rises, define a predictable policy. Options include delaying new prefill, limiting maximum context length, evicting idle sequences, or recomputing cache for selected requests. Recompute can save memory at the cost of extra compute, so it should be treated as a deliberate fallback rather than an invisible side effect.

Implementation and tuning workflow

Start with a representative workload, not a single benchmark prompt. Include short chat turns, long retrieved contexts, bursty arrivals, long outputs, cancellations, and multiple tenants. Replay the workload with a baseline scheduler and then vary one control at a time: chunk size, decode reservation, maximum concurrent prefills, and queue policy.

Plot latency over time, not just aggregate averages. Look for decode stalls immediately after long requests enter the system. Compare p50, p95, and p99 values by request class. Also inspect the relationship between prompt length and TTFT. If long prompts have excellent throughput but interactive p99 degrades, increase decode protection or reduce the prefill quantum. If decode is stable but GPU utilization falls sharply, the chunks may be too small or the scheduler may be reserving more capacity than necessary.

Roll out changes gradually. Use a per-model or per-pool configuration, expose scheduler counters, and keep a fast rollback path. Useful counters include prefill tokens admitted, prefill tokens deferred, decode steps delayed, chunk execution time, queue age, cache allocation failures, and cancellations before first token.

When chunked prefill is not enough

Some workloads need additional isolation. If interactive traffic has strict latency objectives, place it on a dedicated GPU pool or reserve capacity through admission control. If prompts are extremely long, consider prompt caching, retrieval limits, context compression, or an asynchronous workflow. If output generation dominates cost, speculative decoding or model routing may have a larger effect than prefill scheduling.

Chunked prefill is best understood as a scheduling primitive, not a universal performance switch. It works when the serving engine can pause and resume prefill safely, the KV cache is managed incrementally, and the scheduler has an explicit policy for protecting decode. The production objective is not maximum utilization at every moment. It is predictable service: long prompts continue making progress, while users with active streams continue receiving tokens at a stable rate.

Search This Blog

Technology On the Net