Temperature, Seeds and Determinism (What You Can Control)

TL;DR: No. Identical inputs to a hosted LLM do not guarantee identical outputs, even at temperature 0 with a fixed seed. Four things break it: sampling, floating-point non-associativity in batch-variant GPU kernels, mixture-of-experts routing that depends on batch composition, and provider-side infrastructure changes. You can narrow the variance. You cannot eliminate it.

Is AI output ever perfectly deterministic?

Not through a hosted API, no. AI determinism, in the strict sense of identical inputs producing identical outputs, is achievable only if you control the whole stack, and even then it costs you a large share of your throughput.

The advice in circulation says the opposite: set temperature to 0 and the model becomes a pure function. That was never true, and both OpenAI and Anthropic now say so in their own documentation.

OpenAI's published OpenAPI specification, version 2.3.0 (read directly from the openai-openapi repository, accessed August 26, 2026), attaches this to the seed field: "our system will make a best effort to sample deterministically ... Determinism is not guaranteed."

Anthropic is blunter. From the Claude Opus 5 migration guide (accessed August 26, 2026): "If you were using temperature = 0 for determinism, note that it never guaranteed identical outputs on prior models."

Four mechanisms are at work, and they stack. Only the first is about sampling.

Why doesn't temperature 0 make output deterministic?

Because temperature only governs the dice roll, and the dice roll is not the only source of variation. Removing it leaves three others untouched.

At each step the model scores every token in its vocabulary. Temperature rescales those scores before they become probabilities. Push it to 0 and the distribution collapses toward its single most probable token. The randomness of choosing is gone.

What remains is the randomness of computing. Those probabilities are the end of a very long chain of floating-point arithmetic on a GPU, and that chain is not guaranteed to produce bit-identical numbers between runs. When the top two candidates are numerically close, a difference in the last bits is enough to swap them. The model continues from a different token, and every token after that is conditioned on the change.

There is a second problem with reaching for temperature at all in 2026: on Claude you cannot. Anthropic's Messages API reference (accessed August 26, 2026) marks temperature, top_p and top_k deprecated, and states that models released after Claude Opus 4.6 "do not support setting temperature," accepting 1.0 for backwards compatibility and rejecting every other value with a 400. In the Python SDK v1.0 and later the field is not defined at all, so passing it raises a TypeError.

So the most common determinism tip on the internet now returns an HTTP 400 on Anthropic's flagship model. Post 114 has the full parameter table for all three vendors, and post 48 covers what these parameters do.

What does floating-point non-associativity have to do with it?

It is the root cause, and it has a specific, measured consequence: your result depends on how many other people are querying the server at the same moment.

Floating-point addition is not associative: adding a tiny number to a huge one loses precision, so the order you sum a list in changes the total in the last bits. GPU kernels sum in whatever order is fastest for the current shape of the work, so the same operation can return a slightly different number at a different batch size.

The clearest published account is Defeating Nondeterminism in LLM Inference by Horace He at Thinking Machines Lab, published September 10, 2025 and accessed August 26, 2026. Its finding is that the usual explanation is wrong:

the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies

And the consequence, stated exactly:

From the perspective of an individual user, the other concurrent users are not an "input" to the system but rather a nondeterministic property of the system

The numbers are worth quoting. Running Qwen/Qwen3-235B-A22B-Instruct-2507 at temperature 0 with the same prompt 1,000 times, they report: "we generate 80 unique completions, with the most common of these occuring 78 times." The completions were identical for 102 tokens and first diverged at token 103.

An independent group reached the same diagnosis. LLM-42 by Gond, Kamath, Ramjee and Panwar (arXiv:2601.17768, January 2026, accessed August 26, 2026): "this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels whose reduction orders vary with batch size."

Does mixture-of-experts routing break determinism?

It can add a second path to the same problem. This is the part people most often get wrong in both directions: it is neither the main cause nor irrelevant.

Start with what is documented. In a token-choice mixture-of-experts layer, a router sends each token to a small number of experts, and each expert has a fixed buffer. From Switch Transformers by Fedus, Zoph and Shazeer (arXiv:2101.03961, accessed August 26, 2026), section 2.2: "If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection."

Read that with determinism in mind. Whether your token reaches its chosen expert or gets skipped depends on how many other tokens in the batch wanted that expert, and batch composition is set by whoever else is hitting the server.

The overflow is not rare where it has been measured. Mixture-of-Experts with Expert Choice Routing by Zhou et al. (arXiv:2202.09368, accessed August 26, 2026): "we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped."

Three caveats, because this is where confident-sounding writing goes wrong.

The mechanism is not universal. MegaBlocks by Gale, Narayanan, Young and Zaharia (arXiv:2211.15841, accessed August 26, 2026) reformulates the computation so that "our approach never drops tokens."

No frontier provider publishes whether its inference stack enforces expert capacity. None of the three documents its serving internals, so treat capacity-driven token dropping as established in the research literature and unconfirmed for any commercial endpoint.

The batch-size effect explains the observed variance on its own. The Thinking Machines analysis never mentions mixture-of-experts, yet the model it tested is one: Qwen's model card (accessed August 26, 2026) gives 128 experts with 8 activated per token. Fixing the reduction order alone took that model to 1,000 identical completions, with the router untouched.

Which providers support a seed, and what does it promise?

Two of three expose one, and neither guarantees anything. Every cell below was checked against the vendor's own published reference on August 26, 2026.

Three details behind that table.

OpenAI's seed carries two warning labels at once. In the spec it is marked deprecated: true and the description opens "This feature is in Beta." The companion field is going the same way: in the ChatCompletion response object, system_fingerprint is now also marked deprecated: true, while still being the thing the seed documentation tells you to monitor. OpenAI's cookbook on reproducible outputs (accessed August 26, 2026) is candid about the ceiling: "There is a small chance that responses differ even when request parameters and system_fingerprint match, due to the inherent non-determinism of our models."

Anthropic has no seed to set. The Messages API request body documents eighteen fields, including max_tokens, output_config, thinking and the three deprecated sampling parameters. None of them is a seed.

Google's is the plainest and the emptiest. The Gemini API v1beta discovery document, revision 20260823, accessed August 26, 2026, describes it in full as: "Optional. Seed used in decoding. If not set, the request uses a randomly generated seed." No reproducibility claim is attached, in either direction.

Can you pin the model version?

You can pin the weights. You cannot pin the behaviour, and one vendor says so explicitly.

Anthropic's Model IDs and versioning page (accessed August 26, 2026) is the most useful document any provider has published on this. First, it corrects a widespread misreading: dateless IDs such as claude-opus-5 are not floating aliases but "the canonical model ID for that release," mapping to "a single, fixed model snapshot."

Second, it says plainly what pinning does not buy you:

Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time. This infrastructure includes components such as the request router, safety classifiers, and sampling logic. Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed.

That is a provider stating in writing that a pinned model can behave differently next month. For anyone running evals, it is the most important sentence in this post.

The other two differ again. OpenAI's spec publishes dated snapshot IDs for older releases, gpt-5.5-2026-04-23 among them, but in version 2.3.0 the current gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna appear with no dated variant, and any ID ending -chat-latest is explicitly a moving target. Google's model documentation (accessed August 26, 2026) recommends that "most production apps should use a specific stable model," while hedging that "stable models usually don't change."

Pin anyway. It removes one of four variables. Just do not report it as reproducibility.

Does prompt caching change the output?

No, and both vendors that document it say so directly. This one is a genuine non-issue, and it is worth clearing off the list.

Anthropic's prompt caching guide (accessed August 26, 2026): "Prompt caching has no effect on output token generation. The response you receive is identical to what you would get if prompt caching were not used."

OpenAI's prompt caching guide (accessed August 26, 2026): "Prompt caching does not change how the model generates output tokens."

One nuance from OpenAI's page is worth keeping, though it is about routing rather than output: "Keys influence routing; they do not pin requests to a machine or guarantee a cache read hit." A prompt_cache_key is a hint, not a lever.

What can you actually control?

Six things, in descending order of how much they buy you. None of them gets you to bit-identical text on a hosted endpoint.

The first row is the only genuine fix, and only if you run the model yourself. vLLM ships it: per the batch invariance documentation (accessed August 26, 2026), VLLM_BATCH_INVARIANT=1 "ensures that the output of a model is deterministic and independent of the batch size or the order of requests in a batch." It is in beta, needs NVIDIA GPUs at compute capability 8.0 or higher, and has been tested on DeepSeek, Qwen dense and MoE variants, Llama 3, GPT-OSS, Mistral and Phi.

Measure the variance you actually have:

# Measure your own determinism budget. Swap the client for any provider.
# Run it against YOUR prompt, YOUR model, at YOUR time of day — load matters.
import collections
from openai import OpenAI

client = OpenAI()
PROMPT = "Classify this ticket as bug, feature, or question: 'App crashes on export.'"
N = 50

outputs = []
fingerprints = set()

for _ in range(N):
    r = client.chat.completions.create(
        model="gpt-5.5-2026-04-23",   # a DATED id, not a floating alias
        messages=[{"role": "user", "content": PROMPT}],
        seed=42,
        temperature=0,
    )
    outputs.append(r.choices[0].message.content.strip())
    if r.system_fingerprint:
        fingerprints.add(r.system_fingerprint)

counts = collections.Counter(outputs)
print(f"unique outputs: {len(counts)} / {N}")
print(f"most common seen {counts.most_common(1)[0][1]} times")
print(f"distinct system_fingerprints: {len(fingerprints) or 'none returned'}")
for text, n in counts.most_common(5):
    print(f"  {n:>3}x  {text[:80]}")

Run it at 3am and again at 3pm. If the unique count changes, you have just watched other people's traffic move your output.

The Anthropic version drops both fields, because both are rejected:

# Claude: no seed, no temperature, no top_p, no top_k. Pin the model, cap the tokens.
curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-opus-5",
    "max_tokens": 512,
    "messages": [{
      "role": "user",
      "content": "Classify this ticket as bug, feature, or question: App crashes on export."
    }],
    "output_config": {
      "format": {
        "type": "json_schema",
        "schema": {
          "type": "object",
          "properties": { "label": { "type": "string" } },
          "required": ["label"]
        }
      }
    }
  }'

Design for verification, not reproducibility

This is the pivot that actually solves the problem people arrive with. Almost nobody needs identical bytes. They need a result they can trust without reading it.

Those are different goals, and only one is achievable on a hosted API. Reproducibility asks "did I get the same text?" Verification asks "is this output correct?" The second question survives a model swap, a busy server and a router change. The first does not survive lunchtime.

Four moves, in order of payoff.

1. Constrain the shape so only the content can vary. Ask for a schema, not prose. When the response is a JSON object with three named fields, drift shows up as a changed value rather than a reworded paragraph you have to diff by eye. All three vendors enforce structured output at the API level, and post 42 covers the prompt-side version for chat windows.

2. Assert on the fields that matter, not the string. A test comparing the whole response to a golden file fails on the first harmless rewording and then gets muted. A test that checks label is one of three permitted values keeps working across model upgrades.

3. Version the prompt like source code. If you cannot say which prompt produced last Tuesday's output, you cannot tell model drift from your own edit. Post 58 makes the case for treating prompts as versioned artifacts.

4. Log the request, the response, the model ID and the timestamp. Without it, every variance report is unfalsifiable.

Prompt Architects sits on the first of those four, rewriting a loose prompt into a structured one with an explicit Role, Task, Format, Constraints and Tone in under two seconds. That does not make the model deterministic, and nothing will, but a prompt that states its output format leaves far less room to vary in the ways that break your parser. It cannot touch your seed, temperature or model ID; the chat interfaces it plugs into expose none of them. For the parameters you can reach from an API client, post 93 works through what happens when top-p and top-k are both set.

The short version

Identical inputs do not guarantee identical outputs, and no major provider claims otherwise. Temperature 0 removes the sampling draw and nothing else. Batch-variant GPU kernels plus fluctuating server load are the dominant published cause, measured at 80 unique completions from 1,000 identical requests. Mixture-of-experts capacity limits can add a second batch-dependent path, documented in the literature and unconfirmed for any hosted endpoint. Pinning the model fixes the weights, not the serving stack around them.

If you need bit-identical output, self-host and pay about 60% of your throughput for it. If you need trustworthy output, which is what almost everyone actually means, constrain the shape, assert on the fields, version the prompt and log everything.