Speculative Decoding for Production LLM APIs in September 2026: How Draft Models Cut Latency and Cost

Speculative decoding is one of the most practical ways to reduce latency in a production large language model API without changing the user-facing model. Instead of asking a large target model to generate one token at a time, the serving system uses a smaller draft model to predict several tokens ahead. The target model then verifies those predictions in a single forward pass and accepts the matching prefix. When the draft is accurate, the target model produces multiple tokens for roughly the cost and latency of one verification step.

The technique is especially useful for interactive applications, coding assistants, customer-support agents, and structured generation endpoints where time to first complete response matters. It is not a universal optimization, however. The performance benefit depends on the relationship between the draft and target models, the sampling configuration, hardware utilization, prompt length, output length, and the quality of the serving scheduler.

How speculative decoding works

In ordinary autoregressive decoding, the target model generates a token, appends it to the sequence, generates the next token, and repeats. Each step depends on the previous one. A fast GPU can still spend much of its time launching many small, sequential operations rather than performing large, efficient matrix multiplications.

Speculative decoding changes the division of labor:

The draft model generates a short sequence of candidate tokens, often called a draft block.
The target model evaluates that entire block in parallel.
The serving engine compares the target distribution with the draft decisions.
The accepted prefix is committed, and generation resumes from the first rejected token.

Suppose the draft model proposes eight tokens and the target accepts six. The system has advanced six tokens after one target verification pass. If the seventh token is rejected, the target can usually sample a replacement at that position and begin the next cycle. The target model remains responsible for the final output distribution; the draft model is an accelerator, not a substitute for the target.

The main performance metric is the average number of tokens accepted per speculation cycle. A useful approximation is:

effective tokens per target step ≈ 1 + accepted draft tokens

If the draft length is eight and the average accepted prefix is five tokens, the target is advancing about six tokens per verification step. The actual speedup is lower than six times because draft-model execution, memory movement, scheduling, and rejected-token handling still consume time.

Choosing draft-target model pairs

A successful pair has two properties that can conflict: the draft model must be substantially cheaper than the target, but it must also predict the target's next tokens accurately. A tiny unrelated model may be inexpensive but produce so many rejected tokens that speculative decoding becomes slower than ordinary decoding.

For production, evaluate candidate pairs using the same prompts, system instructions, tools, temperatures, and output constraints used by the real API. Useful pairing strategies include:

Same-family smaller model: A compact model trained on the same data or instruction format as the target often produces compatible token choices.
Distilled draft model: A draft specifically trained to imitate the target can provide better acceptance than a general-purpose small model.
Quantized draft: A lower-precision version of a compatible model can reduce draft cost while preserving much of its prediction quality.
Prompt-specialized draft: For code completion, SQL generation, or short support replies, a specialized draft can outperform a general model of similar size.

Tokenizer compatibility is a hard requirement for conventional token-level verification. If the draft and target use different tokenizers, the serving engine needs a more complicated alignment or re-tokenization path, which can erase the benefit. The models should also share compatible vocabulary and chat-template behavior. Differences in stop-token handling, tool-call markers, or JSON delimiters can create unexpected rejection patterns.

Acceptance rates matter more than model size

Draft-model size is an incomplete measure of speculative decoding quality. The more important question is how many proposed tokens the target accepts under production conditions. Measure acceptance at several levels:

Token acceptance rate: The percentage of draft tokens accepted by the target.
Accepted prefix length: The average number of consecutive tokens accepted per cycle.
Cycle gain: The number of output tokens produced per target verification pass.
End-to-end speedup: The latency improvement including draft execution and queueing.

A draft with a high token acceptance percentage can still perform poorly if it produces only short blocks or if its forward pass is expensive. Conversely, a moderate acceptance rate may be worthwhile when the target model is very large and the draft runs efficiently on otherwise idle capacity.

Acceptance is usually highest for predictable text: boilerplate code, repeated formats, short answers, and constrained JSON. It tends to fall for creative writing, ambiguous questions, long reasoning traces, rapidly changing conversation context, and outputs containing many tool calls. Temperature also matters. Higher temperature generally makes target behavior less predictable, while greedy or low-temperature decoding is easier for a draft to imitate.

Medusa- and EAGLE-style draft generation

External draft models are not the only approach. Medusa-style systems add multiple prediction heads to the target model or a closely related model. Instead of running a separate autoregressive draft model, these heads predict several future positions in parallel. This can reduce draft overhead and simplify deployment because the additional heads share the target model's hidden representation.

Medusa-style multi-head drafts are attractive when GPU memory is tight or when maintaining a separate draft model would complicate model loading. Their quality depends on how those heads are trained and how the verifier combines their proposals. They are often effective for common decoding workloads, but they may require target-specific training or checkpoint support.

EAGLE-style methods use richer feature-level predictions rather than relying only on ordinary token-level next-token guesses. The draft component predicts future hidden states or related representations, allowing it to generate longer candidate sequences with fewer full model computations. These methods can deliver strong speedups, particularly when integrated deeply into the inference engine, but they are more sensitive to model architecture, checkpoint compatibility, and implementation details.

For an API operator, the practical distinction is straightforward: an external draft model is easier to swap and benchmark, while multi-head or feature-level methods can reduce overhead but usually require tighter coupling to the target model. Choose the least invasive method that meets the latency target.

When it beats continuous batching alone

Continuous batching improves throughput by combining requests that arrive at different times into efficient batches. It is usually the first optimization to implement for a shared LLM API. Speculative decoding addresses a different bottleneck: the sequential number of target-model decoding steps for each request.

Speculative decoding is most likely to beat continuous batching alone when:

Requests are interactive and sensitive to per-request latency.
Outputs are long enough for decoding time to dominate prompt processing.
The target model has spare compute capacity during decode steps.
The draft model has high acceptance on the application's workload.
Requests are not large enough to keep the target GPU fully saturated through batching.

Continuous batching may be preferable when the service is throughput-bound, acceptance is low, or requests have highly variable generation patterns. Speculation also competes for memory bandwidth and GPU memory. A well-designed production system benchmarks both techniques together rather than treating them as mutually exclusive. Continuous batching can schedule speculative verification batches, but the scheduler must account for variable numbers of accepted tokens.

Important API knobs

Expose operational controls rather than hard-coding one configuration:

Draft model: Select the compatible draft checkpoint or multi-head strategy.
Draft length: Start with a small block, such as four or eight tokens, and increase it only when acceptance remains strong.
Maximum draft length: Cap speculation to prevent waste on unpredictable requests.
Temperature and sampling mode: Use target-compatible sampling and avoid silently changing output semantics.
Acceptance threshold: For probabilistic verification, configure how conservative the acceptance rule should be.
Fallback policy: Disable speculation for tools, schemas, or workloads with consistently poor acceptance.
Adaptive speculation: Adjust draft length from recent acceptance, latency, and queue-depth measurements.
Budget controls: Set limits for draft tokens, draft GPU time, and memory reserved per request.

These knobs should be available through server configuration or internal routing policy. Avoid exposing raw model-engine settings directly to every public API client unless they are part of a stable product contract.

Production pitfalls

The first pitfall is measuring the wrong latency. A benchmark that reports target verification time but excludes draft generation is not representative. Track time to first token, time between visible tokens, total response time, GPU utilization, queue delay, accepted tokens per cycle, and draft overhead.

The second pitfall is assuming exact output equivalence without testing sampling semantics. Correct speculative verification can preserve the target distribution, but implementation errors around random-number streams, top-p filtering, repetition penalties, stop sequences, or grammar constraints can change results. Run statistical and exact-output tests for every supported decoding mode.

Structured output creates another challenge. JSON schemas and grammar-constrained decoding may reject otherwise plausible draft tokens. Some engines can apply the same constraint during drafting; others should disable speculation when constraints become restrictive. Tool calls require special care because a single rejected delimiter or argument token can invalidate the whole draft block.

Memory pressure is easy to underestimate. Serving the target, draft, key-value caches, scheduler metadata, and multiple in-flight requests can reduce the batch size enough to eliminate the expected gain. Measure peak memory, not just model weight size.

Finally, acceptance rates drift. New system prompts, model revisions, customer traffic, and tokenizer changes can all reduce compatibility. Export acceptance and speedup metrics by route, model pair, tenant class, prompt category, and decoding configuration. Set an automatic fallback when speculation increases p95 latency or cost.

A practical rollout plan

Begin with an offline corpus representing real prompts and outputs. Compare continuous batching, speculative decoding, and both together using identical hardware and sampling settings. Then deploy speculation behind a feature flag for a small percentage of traffic. Start with a conservative draft length and disable it for low-acceptance routes.

Promote the configuration only when it improves end-to-end p50 and p95 latency without increasing error rates, output inconsistencies, or cost per completed request. The best production setup is rarely the one with the largest draft block. It is the one that maintains predictable acceptance while fitting cleanly into the scheduler, memory budget, and API's correctness guarantees.

Search This Blog

Technology On the Net