Load Testing Production LLM Endpoints in October 2026: Realistic Token Mixes, TTFT and Inter-Token Latency Percentiles, and Finding the Saturation Point

Most teams find out how much traffic their LLM endpoint can handle on the worst possible day: a launch, a marketing email, or a customer that suddenly batches ten thousand requests. Classic web load testing does not prepare you for that, because an LLM request is not a short, uniform HTTP call. It streams, it varies in cost by orders of magnitude depending on prompt and output length, and it competes for GPU memory with every other request in flight.

This post walks through how to load test an LLM endpoint so the numbers actually predict production: what to measure, how to build a realistic workload, how to generate load without fooling yourself, and how to read the results to find the point where the system tips over. It applies whether you self-host with an inference server like vLLM, SGLang, or TGI, or run a gateway in front of a hosted provider.

Why LLM load testing is different

A typical API load test assumes requests are roughly similar in size and that latency is one number. Neither holds for LLMs:

Cost per request varies wildly. A 200-token prompt with a 50-token answer and a 30,000-token prompt with a 1,500-token answer hit the same URL but consume very different amounts of compute and KV cache memory.
Latency has phases. The prefill phase processes the prompt and determines time to first token. The decode phase generates output one token at a time and determines how smooth streaming feels. They degrade differently under load.
Batching changes everything. Inference servers batch concurrent requests together, so throughput rises with concurrency until memory or compute runs out, and then latency climbs sharply.
Caching skews results. Prefix caching means repeated identical prompts are far cheaper than unique ones. A test that sends the same prompt over and over measures your cache, not your capacity.

The metrics that matter

Collect these per request, then report percentiles rather than averages. Averages hide the tail, and the tail is what users complain about.

Time to first token (TTFT): from sending the request to receiving the first streamed token. This is what makes a chat UI feel responsive or frozen.
Inter-token latency (ITL), sometimes reported as time per output token (TPOT): the gap between streamed tokens after the first. High ITL makes text crawl even when TTFT is fine.
End-to-end latency: total time until the stream closes. This matters most for non-streaming and agent tool calls, where nothing happens until the full response arrives.
Throughput: completed requests per second and output tokens per second across the whole system.
Error rate by type: HTTP 429 rate limits, 5xx errors, timeouts, and streams that close early. Count these separately, because each points to a different bottleneck.
Goodput: the rate of requests that completed successfully and met your latency targets. Raw throughput can keep rising while goodput collapses, which is exactly the failure you want to catch.

Write your targets down before you test, for example "p95 TTFT under 1.5 seconds and p95 ITL under 60 milliseconds for chat traffic." Without a target you cannot say where the saturation point is, only that latency went up.

Build a workload that looks like production

The single biggest mistake is testing with one fixed prompt. Build the workload from real traffic instead.

1. Sample real token distributions

Pull a week of request logs and record input token count and output token count per request. You do not need the prompt text itself, which also keeps personal data out of your test harness. Bucket the counts, or just keep the raw pairs and sample from them during the test. If you have several traffic classes, such as chat, summarization, and agent steps, keep them separate and mix them in their real proportions.

2. Generate synthetic prompts with matching lengths

For each sampled pair, generate a prompt of the right input length and set max_tokens to the sampled output length. To force the model to actually produce that many tokens, some harnesses use an instruction like "continue writing" plus an ignore-end-of-sequence option when the server supports one. Check your server's documentation, since not every API exposes that.

3. Control prefix sharing deliberately

Real traffic often shares a long system prompt. Decide what fraction of test requests share a common prefix and make it match production. Run once with realistic sharing and once with fully unique prompts, so you know both your typical case and your worst case.

4. Model arrivals, not just concurrency

Users do not wait politely for each other. Requests arrive independently, often in bursts. A Poisson arrival process, where the gap between requests is drawn from an exponential distribution, is a reasonable default. Add explicit bursts if your traffic has them, such as cron jobs that fire on the hour.

Open-loop vs. closed-loop load generation

A closed-loop generator runs a fixed number of virtual users, each sending a new request only after the previous one finishes. When the server slows down, the generator automatically sends less traffic, so the test politely backs off at exactly the moment you need it to push. This is a form of the problem often called coordinated omission, and it makes overloaded systems look healthier than they are.

An open-loop generator sends requests on a schedule regardless of how fast responses come back. If the server falls behind, requests pile up, just as they would in production. Use open-loop generation for capacity tests. Closed-loop tests are still useful for answering a narrower question, such as "how does latency look with exactly 32 concurrent streams?"

A minimal open-loop harness

The sketch below sends streaming requests to an OpenAI-compatible chat endpoint at a target rate with Poisson arrivals, and records TTFT, inter-token gaps, and end-to-end latency. It counts each streamed chunk as a token for simplicity; for exact numbers, use the usage data your server returns.

import asyncio, json, random, time
import httpx

URL = "http://localhost:8000/v1/chat/completions"
MODEL = "your-model"
RATE = 4.0          # requests per second
DURATION = 300      # seconds
results = []

async def one_request(client, prompt, max_tokens):
    start = time.perf_counter()
    first, last, gaps, ok = None, None, [], False
    body = {"model": MODEL, "stream": True, "max_tokens": max_tokens,
            "messages": [{"role": "user", "content": prompt}]}
    try:
        async with client.stream("POST", URL, json=body, timeout=120) as r:
            if r.status_code != 200:
                results.append({"status": r.status_code})
                return
            async for line in r.aiter_lines():
                if not line.startswith("data: ") or line == "data: [DONE]":
                    continue
                now = time.perf_counter()
                if first is None:
                    first = now
                elif last is not None:
                    gaps.append(now - last)
                last = now
            ok = True
    except httpx.HTTPError as e:
        results.append({"status": "error", "error": type(e).__name__})
        return
    end = time.perf_counter()
    results.append({"status": 200, "ok": ok,
                    "ttft": (first - start) if first else None,
                    "gaps": gaps, "e2e": end - start})

async def main(workload):
    async with httpx.AsyncClient(limits=httpx.Limits(max_connections=1000)) as client:
        tasks, t_end = [], time.perf_counter() + DURATION
        while time.perf_counter() < t_end:
            prompt, max_tokens = random.choice(workload)
            tasks.append(asyncio.create_task(one_request(client, prompt, max_tokens)))
            await asyncio.sleep(random.expovariate(RATE))
        await asyncio.gather(*tasks)
    with open(f"results_{RATE}.json", "w") as f:
        json.dump(results, f)

Here workload is a list of (prompt, max_tokens) pairs built from your sampled distribution. Run the generator from a machine close to the endpoint with plenty of CPU, and confirm the generator itself is not the bottleneck by checking that the actual send rate matches the target.

You do not have to write your own harness. General tools like Locust and k6 can drive streaming endpoints with some scripting, and inference servers often ship LLM-specific benchmarks; vLLM, for example, includes a serving benchmark command, and projects like GuideLLM focus on this exact problem. Whatever you use, verify that it supports open-loop arrival rates and reports TTFT and inter-token latency separately.

Run a step-load test to find the knee

Do not run one test at one rate. Run a sequence:

Warm up for a few minutes at low load so model weights, CUDA graphs, connection pools, and autoscalers settle. Discard warm-up results.
Step the arrival rate upward, for example 1, 2, 4, 6, 8, 10 requests per second, holding each step long enough to reach steady state. Several minutes per step is a sensible starting point, longer if your output lengths are large.
Record percentiles per step: p50, p95, and p99 for TTFT, ITL, and end-to-end latency, plus error counts and goodput.
Stop when goodput falls or errors climb past what you would accept in production.

Plot latency percentiles against arrival rate. You will usually see a flat region where batching absorbs new load cheaply, then a knee where queues start to form and p95 TTFT climbs steeply. Your safe operating capacity is somewhere below that knee, with headroom for bursts. Many teams plan around a fraction of the knee rate rather than the knee itself, but the right margin depends on how spiky your traffic is.

Reading the results

Different symptoms point to different bottlenecks:

TTFT rises but ITL stays flat: requests are queuing before they start, or prefill of long prompts is the constraint. Look at admission control, chunked prefill settings, or adding replicas.
ITL rises as concurrency grows: decode is getting crowded because batches are large. You may be trading per-user streaming speed for throughput; decide whether that trade fits your product.
Sudden 5xx or out-of-memory errors at a specific load: KV cache capacity is exhausted, often triggered by long-context requests in the mix. Test the long tail of your input distribution on its own.
429s from a hosted provider: you are hitting rate or token-per-minute limits before compute limits. Your capacity is a quota question, so the fix is quota increases, multiple deployments, or smoothing traffic, not more hardware.
Great numbers that do not match production: suspect excessive prefix sharing, too-short outputs, or a closed-loop generator.

Practical guardrails

Never load test a hosted provider casually. You pay for every token, and heavy unannounced traffic may trip abuse controls or affect your account. Check the provider's terms, set a hard spend cap on the test key, and test against a separate project or deployment.
Isolate the test environment. Running a capacity test against shared production GPUs measures your test and hurts real users at the same time.
Test the full path. Benchmark the inference server alone to know its ceiling, then the full path through your gateway, auth, retrieval, and guardrails. The difference tells you where non-model latency lives.
Re-run on every change that matters. A new model version, quantization setting, context length limit, or server upgrade can move the knee. Keep the workload file and harness in version control so runs are comparable.
Include failure cases. Add a step where clients disconnect mid-stream and a step with a burst of maximum-length prompts. Those are the moments production breaks.

A short checklist

Write latency targets for each traffic class.
Sample input and output token lengths from real logs.
Generate synthetic prompts that match those lengths and realistic prefix sharing.
Use an open-loop generator with Poisson arrivals and some bursts.
Warm up, then step the arrival rate and hold each step to steady state.
Report p50, p95, and p99 for TTFT, inter-token latency, and end-to-end latency, plus errors and goodput.
Find the knee, plan capacity below it, and re-test after every significant change.

Done this way, a load test stops being a single impressive throughput number and becomes a map of how your endpoint behaves as traffic grows. That map is what lets you set autoscaling thresholds, rate limits, and launch plans with confidence instead of hoping the GPUs hold.

Search This Blog

Technology On the Net