Sampling Settings and Reproducibility for Production LLM Apps in October 2026: Temperature, Top-p, Seeds, and Why Identical Prompts Still Return Different Answers

Someone on your team files a bug: "The assistant gave a great answer yesterday, and today the exact same question gets a worse one." You rerun it and get a third answer. Nobody changed the prompt. Nobody changed the model. So what happened?

Usually nothing is broken. Language models generate text by sampling, and sampling is random unless you take deliberate steps to narrow it. Even when you do, the serving stack can add small variations of its own. This guide explains what temperature, top-p, top-k, and seeds actually do, why "temperature 0" is not a determinism guarantee, and how to set up a production app so different answers are expected, measured, and debuggable instead of mysterious.

What the model actually does at each step

At every step, the model outputs a score (a logit) for every token in its vocabulary. Those scores are turned into probabilities with a softmax, and the next token is chosen from that distribution. The sampling settings change how that choice is made:

• Temperature divides the logits before the softmax. Values below 1 sharpen the distribution so the top tokens dominate. Values above 1 flatten it so unlikely tokens get picked more often. As temperature approaches 0, sampling approaches greedy decoding: always take the single most likely token.
• Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, then samples from that set. With top_p=0.9, the long tail of unlikely tokens is cut off.
• Top-k keeps only the k most likely tokens. Not every hosted API exposes it, but most open-source serving engines do.
• Seed fixes the random number generator used for sampling, so the same distribution gives the same pick, at least when everything else is equal.

One small change early in a response changes everything after it, because each token becomes part of the context for the next one. That is why two runs can share the first sentence and then end up in completely different places.

Photo: 6sided dice by Diacritica, CC BY-SA 3.0, via Wikimedia Commons

Why temperature 0 still is not fully deterministic

Setting temperature to 0 removes most of the randomness, but many teams are surprised that outputs still drift occasionally. Common reasons:

• Floating-point math is not associative. Adding the same numbers in a different order can give a slightly different result. GPU kernels may sum in different orders depending on how work is split.
• Batching changes the math. On a shared inference server, your request is batched with other people's requests. Different batch sizes can select different kernels or reduction orders, so the logits for your request can differ in the last few digits. When two candidate tokens are nearly tied, that tiny difference flips which one wins.
• Hardware and software versions vary. A provider may route you to different GPU types or roll out a new serving build. The model weights are the same, the arithmetic is not.
• Mixture-of-experts routing. In MoE models, small numerical differences can change which experts handle a token, which can amplify the effect.
• The model behind an alias changed. If you call a floating alias instead of a dated model version, the model itself may have been updated.

Some providers expose a seed parameter and document it as best-effort: same seed and same parameters usually give the same output, but not always. Some also return a backend or system fingerprint field that changes when their serving configuration changes. If your provider offers one, log it; it tells you whether a difference came from your side or theirs.

Also note that some reasoning models fix or ignore sampling parameters entirely, or reject requests that set them. Check the documentation for each model you call rather than assuming every model honors temperature.

Step 1: Choose settings per task, not per app

A single global temperature is a common mistake. Different calls in the same app want different behavior. A reasonable starting point:

• Extraction, classification, routing, tool-argument generation: temperature 0 (or the lowest the model allows). You want the most likely answer every time.
• Question answering over retrieved documents: low temperature, roughly 0 to 0.3. You want faithfulness, not creativity.
• Drafting emails, summaries, or product copy: moderate temperature, roughly 0.5 to 0.8, so outputs do not all sound identical.
• Brainstorming or generating multiple options: higher temperature, roughly 0.8 to 1.0, combined with asking for several candidates.

These are starting points, not rules. Test them against your own evaluation set. Many API docs also recommend changing temperature or top-p, but not both at once, because their effects overlap and tuning two knobs together makes results harder to interpret.

Keep settings in a config object per call type so they are reviewed like code:

SAMPLING = {
    "classify_ticket": {"temperature": 0.0, "max_tokens": 20},
    "answer_with_sources": {"temperature": 0.2, "max_tokens": 800},
    "draft_reply": {"temperature": 0.7, "top_p": 1.0, "max_tokens": 600},
    "brainstorm_titles": {"temperature": 0.9, "n": 5, "max_tokens": 200},
}

Step 2: Log everything you need to replay a call

When someone reports "it answered differently," you need to know whether the inputs were actually identical. Log, for every call:

• The exact model ID (a dated version, not just an alias).
• All sampling parameters, including defaults you did not set explicitly.
• A hash of the fully rendered prompt: system prompt, messages, tool definitions, and retrieved context.
• The seed, if you used one, and any fingerprint field the provider returns.
• The provider region or endpoint, if you route across several.

import hashlib, json

def request_fingerprint(model, messages, tools, params):
    payload = json.dumps(
        {"model": model, "messages": messages, "tools": tools, "params": params},
        sort_keys=True, ensure_ascii=False,
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]

In practice, the most common cause of "same question, different answer" is that the question was not the same. Retrieval returned a different chunk, the conversation history was trimmed differently, or a timestamp was injected into the system prompt. A prompt hash catches all of these in seconds.

Step 3: If you need the same answer, store it

If a business process truly requires that the same input always produces the same output (an audit trail, a generated contract clause, a cached FAQ answer), do not rely on the model to reproduce itself. Store the output keyed by the request fingerprint and return the stored result. Regenerate only when you intend to, such as after a model upgrade, and record that you did.

The same applies to tests. Unit tests that call a live model will be flaky. Record real responses once, save them as fixtures, and replay them in unit tests. Keep live-model checks in a separate evaluation suite that tolerates variation.

Photo: Tabuleiros de Galton (antes e depois), exhibit by Estes Objethos Atelier, photo by Rodrigo Tetsuo Argent, CC BY-SA 4.0, via Wikimedia Commons

Step 4: Evaluate with multiple samples, not one

If your app runs at a temperature above 0, a single evaluation run tells you about one draw from a distribution. A prompt change that looks better on one run may just be luck. Instead:

1. Run each evaluation case several times (for example 5 or 10).
2. Record the pass rate per case, not just pass or fail.
3. Flag cases that pass sometimes and fail sometimes. These are your unstable cases, and they usually point to an ambiguous instruction or a missing piece of context.
4. Compare prompt versions on the average pass rate across the whole set.

def stability_report(case_ids, run_case, samples=5):
    report = {}
    for cid in case_ids:
        results = [run_case(cid) for _ in range(samples)]
        rate = sum(results) / samples
        report[cid] = {"pass_rate": rate, "unstable": 0 < rate < 1}
    return report

Unstable cases are often more useful than outright failures. A case that fails every time is usually a missing capability. A case that passes three times out of five is usually a prompt that leaves room for two different interpretations, and that is something you can fix.

Step 5: Use variation on purpose when it helps

Randomness is not only a problem. For tasks with a checkable answer, you can sample several responses and pick the best:

• Majority vote: for classification or short factual answers, sample three to five times and take the most common answer. Disagreement between samples is also a useful signal that the input is ambiguous.
• Generate and verify: for code or structured output, sample several candidates and keep the first one that passes a validator, test, or schema check.
• Offer options: for creative tasks, show the user two or three variations instead of one.

from collections import Counter

def majority_label(classify, text, k=5):
    votes = Counter(classify(text) for _ in range(k))
    label, count = votes.most_common(1)[0]
    return label, count / k   # agreement ratio

Each extra sample costs another call, so reserve this for decisions where a wrong answer is expensive, and keep k small.

Step 6: Explain the behavior to users and support staff

If your product has a "regenerate" button, users already expect variation. If it does not, a different answer to the same question can look like a bug. Two small things help:

• For factual or policy answers, keep temperature low and ground answers in retrieved sources, so the substance stays stable even when wording changes.
• Give support staff a way to look up the request fingerprint for a reported conversation, so they can tell at a glance whether the inputs differed.

A quick checklist

• Sampling settings live in versioned config per call type.
• Deterministic tasks run at temperature 0, with a seed where the provider supports it.
• Every call logs a dated model ID, all parameters, a prompt hash, and any provider fingerprint.
• Outputs that must never change are stored, not regenerated.
• Unit tests replay recorded fixtures; live evaluations use several samples per case.
• Unstable evaluation cases are reviewed and the prompt is tightened.

Treat model output as a distribution, not a single fixed value. Once you log the inputs, pick settings per task, and measure stability across samples, "it answered differently" turns from a mystery into a quick lookup.