Speculative Decoding for Faster LLM Inference in Production

Speculative decoding is an inference technique that can reduce the latency and cost of large language model serving without changing the target model’s output distribution. Instead of asking a large model to generate every token sequentially, a smaller draft model quickly proposes several tokens. The larger target model then verifies those tokens in a single forward pass and accepts the prefix that it considers valid. The process repeats until the response is complete.

The technique is attractive because autoregressive decoding is often limited by sequential latency rather than raw computation. A target model may spend one expensive forward pass producing just one next token. Speculative decoding uses the draft model to make multiple proposals between target-model checks, allowing the target model to process several positions in parallel.

The basic draft-and-target algorithm

A speculative decoding system has two models:

The target model is the model whose output is ultimately served. It is usually larger, more capable, and more expensive to run.
The draft model is smaller and faster. Its job is to predict a short sequence of likely future tokens, not to produce the final answer independently.

Assume the system has already generated a prompt plus a response prefix. The draft model generates a block of several candidate tokens, such as four or eight. The target model then evaluates the original prefix followed by the proposed block. Because transformer inference can evaluate multiple new positions in one pass, verification is cheaper than generating each candidate token through separate sequential target-model calls.

The target model examines the draft distribution at each proposed position. In the simplest greedy version, a draft token is accepted when it matches the token the target model would select. Verification stops at the first mismatch. The target model’s selected token is emitted at that position, and the remaining draft tokens are discarded. If every draft token is accepted, the system may also sample or select one additional token from the target distribution, depending on the implementation.

For probabilistic sampling, acceptance is more subtle. The verifier must preserve the target model’s distribution rather than simply accepting tokens that look plausible. A common approach uses rejection sampling: a draft token is accepted according to a probability derived from the target probability divided by the draft probability. If a token is rejected, a residual distribution is used to choose the replacement. This preserves the target distribution when implemented correctly.

Why the method can be faster

Plain autoregressive serving performs one target-model decoding step per output token. Each step includes reading model weights, running attention for the new position, and updating the key-value cache. The target model may be highly optimized, but the dependency between token positions remains sequential.

Speculative decoding changes the balance. The draft model performs several cheap sequential steps, while the target model verifies a block in one parallelized operation. The target model still does substantial work, but fewer target-side decoding iterations are required.

The benefit depends on three variables:

Draft cost: how much time the draft model needs to produce a block.
Acceptance rate: how many proposed tokens survive target verification.
Target verification cost: how efficiently the serving stack processes multiple candidate positions.

A fast draft model with poor agreement may provide little benefit because most proposals are rejected. A highly compatible draft model may produce a strong speedup even if it is not extremely small. The best draft model is therefore not always the smallest available model. It is the model that minimizes total time per accepted token.

Acceptance rate is necessary but insufficient

Acceptance rate is usually reported as the fraction of draft tokens accepted by the target. If a draft proposes eight tokens and the verifier accepts six on average, the mean accepted length is approximately six tokens per verification cycle. That is useful, but it does not directly equal the production speedup.

Measure at least these quantities:

Draft generation time per proposal block.
Target verification time per block.
Number of proposed tokens per block.
Accepted tokens per block, including any target-generated replacement token.
End-to-end time to first token.
End-to-end time per output token after the first token.
GPU utilization, memory usage, and energy or cost per generated token.

A simple approximation is:

effective throughput ≈ accepted tokens per cycle ÷ (draft time + verification time)

Compare this with the target model’s plain decoding throughput under the same prompt lengths, batch sizes, quantization settings, and concurrency. Do not compare speculative decoding at batch size one against plain serving at a different load profile. The target model may already benefit from continuous batching, paged attention, CUDA graphs, or optimized kernel fusion.

Acceptance also varies by workload. Code completion, structured JSON, and repetitive text often have high agreement between related models. Creative writing, long-form reasoning, multilingual text, and prompts with unusual terminology may have lower agreement. Report acceptance rates by route, model, language, prompt class, and output length instead of relying on one global average.

Choosing a draft model

The draft and target models should normally be related. They may share a tokenizer, architecture family, vocabulary, or training data. A draft model from the same family often predicts the target model’s likely tokens more accurately than an unrelated model with a similar parameter count.

Important selection criteria include:

Tokenizer compatibility: mismatched tokenization complicates verification and can reduce the practical benefit.
Model residency: both models should remain available without causing memory pressure or frequent eviction.
Draft latency: benchmark the draft under the actual context lengths and concurrency you will serve.
Agreement with the target: evaluate accepted tokens, not only perplexity or standalone quality.
Hardware placement: determine whether both models share a GPU, use separate GPUs, or run with the draft on a CPU or accelerator.

Running both models on one GPU can reduce transfer overhead but create contention. Separate devices can improve overlap, but inter-device communication and synchronization may offset the gain. For smaller targets, the extra draft model may consume too much memory relative to the saved computation.

When speculative decoding beats plain serving

Speculative decoding is most likely to win when the target model is expensive, output generation is long enough to amortize setup costs, and the draft model has high agreement with the target. It is particularly useful for interactive completion, code generation, agent responses, and other workloads where decode latency dominates prompt processing.

It may not help when responses are very short. The initial draft setup, synchronization, and verification overhead can exceed the savings from eliminating only a few target steps. It can also lose at high batch sizes if the serving engine already keeps the target GPU saturated. In that situation, speculative decoding may add draft-model work without reducing the limiting resource.

Long prompts are not automatically a reason to use speculation. Prompt ingestion and prefill can dominate total latency, while speculative decoding primarily accelerates the decode phase. Measure time to first token separately from time between tokens.

Common failure modes

Low acceptance from model mismatch

A draft model may be fast but poorly aligned with the target. Low acceptance increases wasted verification work and can make the system slower than plain decoding. Test representative production traffic before enabling speculation globally.

Draft-model contention

If the draft and target compete for memory bandwidth, tensor cores, or scheduling capacity, the target verification pass may slow down. Monitor both models independently and test under realistic concurrency.

Incorrect sampling behavior

Greedy verification is easier to implement, but sampled generation requires distribution-preserving acceptance and rejection logic. An implementation that simply accepts or rejects tokens using an ad hoc rule can silently alter temperature, top-p, or top-k behavior. Validate output distributions and random-seed behavior against a trusted reference.

KV-cache and memory pressure

Speculative systems may need cache state for the target and draft paths, plus temporary buffers for proposed tokens and verification logits. Context growth can expose memory problems that are not visible in short benchmarks. Test near the maximum supported context length and include concurrent requests.

Unsupported decoding features

Grammar-constrained decoding, tool calls, logit bias, forced prefixes, watermarking, and custom stopping rules can complicate verification. Every feature that changes token eligibility must be applied consistently to both draft and target logic. If that is not possible, route the request to plain serving.

Deployment recommendations

Start with speculative decoding behind a feature flag and keep a plain-serving fallback. Record proposal length, accepted length, rejection position, draft latency, verification latency, queue time, and total decode time. Do not log raw prompts or responses merely for performance measurement; collect aggregate metrics and apply the same privacy controls used elsewhere in the inference stack.

Use adaptive speculation when possible. The system can reduce the proposal length after repeated early rejections and increase it when acceptance remains high. A fixed block size is easier to operate, but it may perform poorly across mixed workloads.

Benchmark at several concurrency levels and with the same batching policy used in production. Include cold starts, model loading, cache fragmentation, request cancellation, streaming delivery, and preemption. Verify that cancellation releases draft and target resources promptly.

Finally, optimize for user-visible latency rather than acceptance rate alone. A configuration with a lower acceptance rate can still win if its draft model is substantially faster or its verification kernels are better optimized. The production decision should be based on measured time to first token, steady-state inter-token latency, throughput, cost, and reliability compared with a well-tuned plain-serving baseline.

Search This Blog

Technology On the Net