Why Do I Get a Different Answer Every Time?

TL;DR: Because hosted AI models sample, and because your request is batched with other people's requests in a way that changes the arithmetic. Anthropic says temperature 0 is still not fully deterministic. Google says a temperature of 0 is deterministic. Both statements are live today. Prompt structure narrows the variance further than any setting you can reach.

If AI gives you different answers each time you run the same prompt, you are not doing anything wrong. You paste the prompt in twice and get two different answers. Sometimes the difference is cosmetic. Sometimes the second run drops the one section you needed, or reorders your table, or decides this time that a caveat is required. It feels like a bug, and the standard advice online is to set the temperature to zero.

That advice is now wrong in three separate ways, and the honest explanation depends on which model you are talking to.

Do the AI companies agree on what causes this?

No. Two of the largest vendors publish directly opposite statements, and both were live when this post was written.

Anthropic's Messages API reference, in the temperature field description, says: "Note that even with temperature of 0.0, the results will not be fully deterministic." Google's Gemini prompting-strategies guide says: "A temperature of 0 is deterministic, meaning that the highest probability response is always selected."

Those cannot both be complete descriptions of the same phenomenon. The gap is not a typo. Google is describing the sampler, which at zero does become a deterministic rule: always take the highest-probability token. Anthropic is describing the system, in which the probabilities themselves are computed by hardware whose arithmetic is not perfectly stable. Both statements are defensible. Neither one tells a user what to expect on its own.

Anthropic also disagrees with itself on a smaller point. The API reference says temperature is unsupported on "Models released after Claude Opus 4.6"; the Opus 5 migration guide says the 400 error applies "on Claude Opus 4.7 or later models". Same rule, two boundaries, two live pages.

Can I just set the temperature to zero?

Increasingly, no. Temperature is the parameter that controls how much randomness enters token selection, and for years the standard fix was to turn it down. Three things changed.

Anthropic removed it. The Opus 5 migration guide states that setting temperature, top_p or top_k to any non-default value on Claude Opus 4.7 or later returns a 400 error, and adds a line worth reading twice: "If you were using temperature = 0 for determinism, note that it never guaranteed identical outputs on prior models." The same guide says prompting is now the recommended way to steer behaviour.

DeepSeek accepts it and ignores it. Its thinking-mode guide says thinking mode does not support temperature, top_p, presence_penalty or frequency_penalty, and that "for compatibility with existing software, setting these parameters will not trigger an error but will also have no effect." Thinking mode is on by default. So the parameter appears to work, returns 200, and does nothing.

Google advises against moving it. The same page that calls temperature 0 deterministic also recommends keeping temperature, top-p and top-k at their defaults on Gemini 3.x models, warning that changes can cause looping or degraded performance on reasoning-heavy tasks.

And if you are using ChatGPT, Claude or Gemini through the consumer app rather than an API, the parameter is not exposed to you at all. There is no slider to find.

Why does temperature zero still drift?

This is the genuinely interesting cause, and almost nothing on page one of the search results explains it.

Computers add floating-point numbers in a fixed number of bits, which means addition is not associative: (a + b) + c does not always equal a + (b + c). Change the order of a large sum and you can change the last bit of the answer. A language model's forward pass is millions of such sums.

The obvious guess is that GPUs race, and whichever core finishes first wins. Thinking Machines Lab tested that guess in "Defeating Nondeterminism in LLM Inference", published 10 September 2025, and found it mostly wrong. Most inference kernels are run-to-run deterministic; run the same kernel twice with the same input and you get the same output. The real culprit is that those kernels are not batch-invariant. The result for your request depends on how many other requests were processed alongside it.

That is the sentence to hold on to. Hosted APIs batch incoming traffic for efficiency, and batch size tracks server load. As the article puts it: "From the perspective of an individual user, the other concurrent users are not an 'input' to the system but rather a nondeterministic property of the system." Their conclusion is blunt: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies."

The measurement is the part people remember. Sampling 1,000 completions at temperature 0 from Qwen3-235B-A22B-Instruct-2507 with the prompt "Tell me about Richard Feynman" produced 80 unique completions. The first 102 tokens were identical every time. At token 103, after all 1,000 runs wrote "Feynman was born on May 11, 1918, in", 992 continued with "Queens, New York" and 8 continued with "New York City". One near-tie, and the two continuations diverge from there. With batch-invariant kernels enabled, all 1,000 runs matched.

A peer-facing paper reaches the same root cause from a different direction. "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference" (arXiv:2506.09501) reports that under bfloat16 with greedy decoding, DeepSeek-R1-Distill-Qwen-7B varied by as much as 9% in accuracy and 9,000 tokens in response length purely from differences in GPU count, GPU type and evaluation batch size.

Did the model change underneath me?

Often, yes, and this is the cause people most reliably misdiagnose as their own error.

OpenAI's prompt engineering guide says it plainly: "Even different snapshots of models within the same family could produce different results." Its recommendation is to pin production applications to a specific snapshot such as gpt-4.1-2025-04-14, and to build test and evaluation suites that measure prompt behaviour when you change or upgrade model versions.

Anthropic approaches the same problem from the opposite end. Its versioning documentation states that every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on, and that Anthropic does not update the weights of an existing model ID. But the same page adds a caveat worth knowing: "Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time," including the request router, safety classifiers and sampling logic, and those updates can "produce minor differences in observable behavior even when the model ID and weights have not changed."

Pinning is necessary. It is not sufficient. And in a consumer app, where you cannot pin anything, a quiet model update is a perfectly ordinary explanation for "it used to do this correctly".

Does web search or file upload make it worse?

Yes, and this one is not subtle.

When a model searches the web, the retrieved text becomes part of the input. Two runs a day apart are not the same request, because the pages changed. OpenAI's web search documentation adds a second layer: "the model can choose to search the web or not based on the content of the input prompt." One run may be grounded in fetched sources; the next may answer from parameters alone. Those two answers were never going to match.

The same applies to file search, connectors, and any tool whose output is not frozen. If you want comparable answers across runs, paste the source text into the prompt rather than asking the model to go and find it. That converts a moving input into a fixed one.

Mixture-of-experts architectures add one more path, though it is less commonly the dominant cause. In token-choice routing each expert has a fixed capacity, and the original Switch Transformer paper describes what happens on overflow: "If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection." Whether your token overflows depends on which other tokens shared the batch. If you want that mechanism unpacked properly, post 127 on temperature, seeds and determinism goes deeper, alongside the parameter explainer for temperature, top-p and top-k.

How do I write a prompt that repeats?

Here is the useful reframe. You cannot remove the model's variance, but most of what you actually notice between two runs is not numerical noise. It is the model making a judgement call you never made for it. Every vague word in your prompt is a decision you delegated, and delegated decisions are exactly the parts that move.

Anthropic's own consistency guidance points the same way: specify the output format precisely, constrain with examples, ground answers in supplied material, and split complex tasks into smaller ones. The patterns below are that advice made copy-pasteable.

1. Give the exact output skeleton

Answer using this template exactly. Do not add, remove, or rename sections.

## Verdict
[one sentence, max 25 words]

## Evidence
- [fact] — [source line number]
- [fact] — [source line number]
- [fact] — [source line number]

## Risk
[one sentence starting with "The main risk is"]

2. Replace every adjective with a number

Rewrite the paragraph below.

Length: 90-110 words. Count them.
Sentences: between 5 and 7.
Sentence length: no sentence over 22 words.
Paragraphs: exactly 1.
Do not add a closing summary sentence.

Words like "concise", "short" and "detailed" are re-interpreted on every run. Numbers are not. This is the single highest-yield change, and it is also the fix for answers that quietly ignore your word count.

3. Name the fields and forbid extras

Return exactly these keys, in this order, and no others:
title, audience, primary_claim, supporting_claims, objection, cta

If a value is unknown, use the string "unknown". Never invent a value.
Never add commentary before or after the object.

4. State the decision rule, not the goal

Rank these 8 headlines.

Rank by, in this priority order:
1. Contains a concrete number
2. Under 60 characters
3. Names the reader's role in the first 3 words

Ties broken by shorter length. Output the ranking as a numbered list only.

5. Fix the ordering explicitly

Sort the output alphabetically by the "name" field, case-insensitive.
Where two names match, sort by "created_at" ascending.
Do not reorder based on importance or relevance.

6. Freeze the vocabulary

Use only these terms for the product tiers: Free, Pro, Advanced, Team.
Do not substitute synonyms such as "basic", "premium", "enterprise", or "plus".
If a tier is not in that list, write "not applicable".

7. Supply the source and close the door

Answer only from the text between <source> and </source>.
If the answer is not in the source, reply exactly: "Not in the source."
Do not use prior knowledge. Do not infer. Do not search.

<source>
[paste your text here]
</source>

8. Anchor the format with one worked example

Convert each note into a task line.

Example input:  "chase invoice from Acme, they're 3 weeks late"
Example output: "Acme | invoice follow-up | overdue 21d | owner: finance"

Now convert these, using exactly that four-field pipe format:
[your notes]

A single demonstration pins the shape more reliably than three sentences describing it, which is the practical case for showing an example rather than describing one.

9. Pre-empt the ambiguity you know is there

If the input mentions more than one date, use the earliest.
If the currency is not stated, assume USD and add a "currency_assumed": true flag.
If the text is under 20 words, return "insufficient input" rather than guessing.

10. Ask for the assumptions, separately

Before your answer, output an ASSUMPTIONS block listing every judgement call
you had to make, one per line, in the form "assumed X because Y".
Then output the answer.

You will see, in writing, the exact places the two runs diverged. Fix those lines in the prompt and the divergence usually goes with them.

11. Ship the instruction block as a fixed header

### OUTPUT CONTRACT v3 — do not modify
Audience: technical buyer, 5+ years experience
Voice: direct, no hype adjectives, British spelling
Never use: "in today's landscape", "leverage", "robust"
Always: cite the source line for any number
Format: markdown, H2s only, no H3s
### END CONTRACT

Paste it above every request in a workflow. It removes the variance that comes from re-describing your standards slightly differently each time. This is the same reasoning behind keeping a brand-voice context you reuse rather than restating it.

12. Make the model check itself against the contract

Produce the answer. Then, in a separate CHECK block, verify it against each
rule below and write PASS or FAIL for each, quoting the offending text on FAIL:
- word count between 90 and 110
- exactly 3 bullets under Evidence
- no sentence longer than 22 words
If any check fails, output the corrected version below the CHECK block.

13. Turn off the discretionary extras

Do not add a preamble, a summary, a disclaimer, a caveat about limitations,
or an offer to help further. Output only the requested artefact.

Those optional flourishes are among the most variable parts of any answer, which is why the disclaimers appear on some runs and not others.

14. Test for variance instead of hoping

Run this prompt 3 times, then compare:
- Are the field names identical across all 3?
- Are the numeric values identical?
- Is the section order identical?
Report only the fields that differed.

Do this once for any prompt you plan to reuse. It takes two minutes and tells you which parts of your output are actually load-bearing versus decorative.

What works if you are calling the API?

Four things, in order of return on effort.

Pin the snapshot. OpenAI recommends it explicitly. It removes the largest and least visible source of drift, which is the model changing while your prompt does not.

Use structured outputs, not a polite request for JSON. OpenAI describes Structured Outputs as a feature that "ensures the model will always generate responses that adhere to your supplied JSON Schema", and separately warns that plain JSON mode "will not guarantee the output matches any specific schema, only that it is valid and parses without errors." Anthropic's consistency guide routes the same way, recommending structured outputs over prompt engineering when you need guaranteed schema compliance. A structured output does not make the prose stable, but it makes the fields stable, which is usually what your code cares about. How JSON prompting changes the failure mode covers the prompt-side version.

Use a seed only if you know what it promises. OpenAI's API specification says the system "will make a best effort to sample deterministically" and then states outright: "Determinism is not guaranteed." Google documents seed as the "seed used in decoding" and notes a random one is used if you omit it, with no guarantee published either way. Anthropic's Messages API does not expose a seed at all, and DeepSeek does not list one on its chat completion request page. A seed narrows variance. Nobody sells it as elimination.

Build an eval suite instead of spot-checking. This is OpenAI's own second recommendation next to snapshot pinning: build tests that measure prompt behaviour so you can monitor performance when you change or upgrade model versions. Ten fixed inputs with asserted properties will tell you more in one run than fifty manual re-rolls. It is the same discipline as treating prompts like something you test rather than something you tweak.

What none of this fixes

Nothing available to a normal user makes a hosted model fully reproducible. Not temperature zero, which two vendors describe in opposite terms and several current models no longer accept. Not a seed, which every vendor that offers one describes as best-effort. Not pinning, which fixes the weights and leaves the serving stack free to change.

Full reproducibility currently means running your own inference stack with batch-invariant kernels, which the Thinking Machines work demonstrates is achievable and costs throughput. That is a reasonable trade for a research lab reproducing a result. It is not a realistic answer for someone who wants their weekly report to come out the same shape twice.

The workable goal is narrower and more useful: make the parts you depend on stable, and stop depending on the parts you cannot control. Field names, counts, ordering and vocabulary can be nailed down with the patterns above. Exact sentence-level wording cannot. Build the workflow so that a reworded sentence is a cosmetic difference rather than a broken pipeline.

The short version

You get different answers because generation samples from a distribution, and because your request shares a batch with strangers whose traffic changes the arithmetic underneath it. Vendors disagree in writing about whether temperature zero fixes the first part, and none of them claim it fixes the second. Model snapshots move, infrastructure changes, and any tool that fetches live data guarantees a different input.

Spend your effort on the output contract rather than the sampling settings. Every number you specify, field you name, and judgement call you make yourself is one fewer thing the model gets to decide differently next Tuesday.