Quantization for Production LLM Serving in September 2026: AWQ, GPTQ, and FP8 Without Guessing Your Latency Budget

Quantization is one of the few serving changes that can cut GPU memory and raise tokens-per-second without rewriting your application. It is also easy to get wrong: pick the wrong format, skip calibration, or ignore decode-path kernels, and you trade a small memory win for quality regressions that show up only on long, tool-heavy prompts. This guide walks through AWQ, GPTQ, and FP8 the way a production team should evaluate them in September 2026—by workload shape, not by blog benchmarks.

Image: mikemacmarketing / photo on flickr via Wikimedia Commons (CC BY 2.0)

Image: Midjourney; prompt suggested by Grok via Wikimedia Commons (Public domain)

What quantization actually changes in a serving stack

A decoder-only transformer spends most of its per-token cost on matrix multiplies against large weight tensors and on reading/writing the KV cache. Quantization shrinks those tensors so more concurrent sequences fit in GPU HBM, or so a single request pays less memory bandwidth per generated token. The important distinction is weight-only versus activation/KV quantization:

• Weight-only INT4/INT8 (common AWQ and GPTQ deployments) keeps activations in higher precision. Prefill and decode still use FP16/BF16 math unless the kernel fuses dequantization into the matmul.
• FP8 weights and activations (Hopper/Blackwell-class kernels) can accelerate both compute-bound prefill and bandwidth-bound decode when the stack has end-to-end FP8 support.
• KV-cache quantization (often INT8 or FP8) is a separate lever. It reduces the memory that grows with batch size and context length, which matters more for long-context agents than for short chat completions.

If your bottleneck is HBM capacity under continuous batching, weight + KV savings usually dominate. If you are compute-bound on long prefills, FP8 activations may matter more than squeezing weights from INT8 to INT4.

AWQ vs GPTQ vs FP8: pick by failure mode, not brand

AWQ (Activation-aware Weight Quantization) protects channels that have large activation magnitudes during calibration. In practice it tends to be robust for chat and instruction-tuned models when you can run a short calibration pass on representative prompts. GPTQ solves a layer-wise least-squares problem and can match or beat AWQ depending on model family and calibration set, but it is more sensitive to how well that set matches production traffic. Neither method invents accuracy; both approximate weights, so your evaluation set must include the hard cases you care about—tool-argument JSON, code edits, multilingual queries, and long RAG contexts.

FP8 is different. On hardware with native FP8 tensor cores and a serving stack that keeps the pipeline in FP8 (not constant cast-to-BF16 round trips), you get both smaller weights and faster matmuls. The catch is ecosystem completeness: tokenizer → attention → MoE experts → sampler must all cooperate. A “FP8 checkpoint” that silently dequantizes every layer on every step can look smaller on disk and still run like BF16.

A practical selection rule for September 2026:

1. Start with the vendor/runtime’s recommended weight format for your GPU generation if you need a same-week rollout.
2. Prefer AWQ or GPTQ INT4 when HBM is the limiter and you can afford a calibration + offline eval loop.
3. Prefer FP8 when you already run on GPUs with solid FP8 kernels and prefill latency is the SLO you miss most often.
4. Treat KV-cache dtype as its own experiment; do not assume weight quantization automatically fixes long-context batching.

A concrete evaluation harness (no vanity metrics)

Do not accept a single MMLU delta as a ship gate. Build a small, boring harness that mirrors production:

• Task suites: short factual Q&A, long-document RAG answers, structured tool calls (valid JSON / schema adherence), and multi-turn agent traces with tool results injected back into context.
• Serving metrics: time-to-first-token (TTFT), inter-token latency (ITL) at target concurrency, max stable QPS before queue growth, and KV memory per active sequence.
• Quality metrics: exact-match or rubric scores where you have labels; for tool calls, schema validity + argument correctness; for RAG, citation presence and groundedness checks against retrieved chunks.
• Regression protocol: freeze prompts and seeds; compare BF16/FP16 baseline vs candidate quantized weights on the same GPU SKU and same continuous-batching config.

Ship only when quality stays within your product tolerance and the serving win is large enough to matter—for example, fitting one more replica’s worth of concurrency on the same GPU, or clearing a TTFT SLO that BF16 misses at peak. If quality holds but latency barely moves, you quantized the wrong bottleneck.

Calibration and data that actually represent traffic

Calibration sets that look like Wikipedia paragraphs will under-stress production agents. Include:

• Real prompt templates with system instructions and tool schemas.
• Retrieved RAG chunks at the lengths you actually pack (not toy 200-token snippets).
• Tool-result payloads: JSON, HTML fragments, stack traces—whatever your agents paste back into the context window.
• Edge languages and code languages you support in production.

Re-run calibration when you change chat templates, add tools, or jump major model versions. A weight file calibrated on last quarter’s traffic is a silent quality footgun.

Runtime checklist before you flip production traffic

Quantized checkpoints fail in ops more often than in papers. Before canarying:

1. Kernel path: Confirm the serving engine loads fused quantized kernels for both prefill and decode. Watch for logs that mention fallback dequantization to FP16 every step.
2. Continuous batching: Re-tune max batch size / max sequenced tokens. Memory headroom from smaller weights often lets you raise concurrency—but attention and KV still scale with live tokens.
3. Speculative decoding interaction: If you use a draft model, quantize draft and target consistently or measure acceptance rate again. A quantized target with an FP16 draft can change reject rates.
4. LoRA / multi-adapter: Verify adapters apply correctly on quantized bases. Some stacks require adapters in a matching precision or merge-then-quantize workflows.
5. Observability: Tag requests with model_variant (e.g., llama-70b-awq-int4) so quality and latency regressions attribute to the right artifact.

When not to quantize aggressively

Stay on BF16/FP16 (or a mild INT8) when:

• Your model is already small relative to GPU memory and you are latency-bound on network or tool I/O, not HBM.
• Outputs are high-stakes and lightly reviewed (medical summarization, financial advice, legal drafting) and you lack a strong eval suite.
• You rely on rare long-tail behaviors—obscure APIs, uncommon languages—that your calibration and eval sets do not cover.
• The serving stack’s quantized path is immature for your architecture (some MoE layouts, unusual attention variants, or custom layers).

In those cases, prefer horizontal scaling, better caching, or a cascade to a smaller model for easy traffic over a rushed INT4 cutover.

A rollout pattern that survives Monday morning

Use a staged path:

1. Offline: Produce the quantized artifact; run the harness against baseline; store scores next to the checkpoint hash.
2. Shadow: Serve quantized weights on mirrored traffic without returning those answers to users; compare latency histograms and automated quality proxies.
3. Canary: Send 1–5% of live traffic; alert on schema-invalid tool calls, elevated user retries, and TTFT/ITL regressions.
4. Promote or revert: Keep the previous artifact immutable and one config flip away. Quantization rollbacks should be faster than model training rollbacks.

Document the decision in one place: GPU SKU, dtype, calibration commit, eval scores, and the concurrency/SLO numbers that justified the change. Future you will need that when the next model drop lands.

Bottom line for September 2026

AWQ and GPTQ remain the workhorse path when memory is the constraint and you can invest in calibration plus a production-shaped eval suite. FP8 is the better default when your hardware and serving stack genuinely keep compute in FP8 and prefill cost dominates. KV-cache quantization is worth a separate experiment for long-running agents. Treat quantization as a capacity and latency feature with a quality budget—not as a free compression checkbox—and you will ship wins you can defend when dashboards spike.