Top-p vs Top-k: What Happens When You Set Both

TL;DR: Top-p and top-k are two filters applied to the same probability distribution, and both keep a prefix of the same sorted token list. The surviving candidate pool is the intersection, which is always the shorter prefix. So the more restrictive filter governs, and which one that is changes at every single token.

If both nucleus sampling (p=0.9) and top-k (k=40) are applied, what is the effective sampling pool?

The intersection of the two filters, which in practice means whichever filter is more restrictive at that step. This is correct, and the reason is worth understanding rather than memorising.

At each generation step the model produces a probability over its entire vocabulary of tokens. Both filters begin by sorting that vocabulary in descending probability order. Top-k keeps the first k entries. Top-p keeps the shortest run of entries whose probabilities sum to at least p.

Both operations therefore return a prefix of the same sorted list. And the intersection of two prefixes of one list is simply the shorter prefix. That is the whole mechanism. There is no clever interaction, no averaging, no negotiation between the parameters.

What that does mean is that neither parameter is reliably the binding one. With k=40 and p=0.9, the top 40 tokens might hold 99% of the probability mass at one step, in which case top-p cuts the pool down to a handful. At the next step the distribution might be flat enough that reaching 90% of the mass takes 200 tokens, in which case k=40 is the constraint. The answer flips within a single sentence.

What is the actual difference between top-p and top-k?

Top-k truncates by count. Top-p truncates by probability mass. That is the entire distinction, and everything else follows from it.

The definitions come from Holtzman et al., The Curious Case of Neural Text Degeneration (arXiv:1904.09751, v1 April 22, 2019, revised February 14, 2020 — accessed August 25, 2026), the paper that introduced nucleus sampling. In the paper's notation, the top-k vocabulary is "the set of size k which maximizes" the summed probability of its members. The top-p vocabulary is "the smallest set such that" the summed probability of its members is at least p.

The paper's stated motivation for nucleus sampling is exactly the weakness of a fixed count. It argues that "the use of a constant k is sub-optimal across varying contexts," and that with nucleus sampling "the size of the sampling set will adjust dynamically based on the shape of the probability distribution at each time step."

In both cases the surviving probabilities are rescaled to sum to 1 before a token is drawn. Hold onto that detail. It becomes the whole story in the last section.

If you want the broader explainer covering temperature alongside these two, that is post 48 on temperature, top-p and top-k. This post stays on the intersection question.

Worked example: k=40 and p=0.9 on the same distribution

Here is a deliberately small, made-up distribution — a toy 10-token vocabulary, not a measurement from a real model. Real vocabularies run past 100,000 tokens, but the arithmetic is identical and easier to follow at this size.

Apply k=40 alone. The vocabulary only has 10 tokens, so all 10 survive. On a real vocabulary you would keep exactly 40. Either way, top-k has removed nothing that top-p would have kept here.

Apply p=0.9 alone. Walk the cumulative column until it reaches 0.9. Rank 4 gets you to 0.870, which is short. Rank 5 gets you to 0.920, which clears the threshold. The nucleus is ranks 1 through 5 — five tokens.

Apply both. Top-k keeps ranks 1–10. Top-p keeps ranks 1–5. The intersection is ranks 1–5. Top-p is the binding constraint, and k=40 did nothing.

Now change one number. Set k=3 and leave p=0.9:

• Top-k keeps ranks 1–3.
• Top-p still keeps ranks 1–5.
• The intersection is ranks 1–3. Top-k is now binding, and p=0.9 does nothing.

And now change the distribution instead. Take a peaked step where the model is confident — say probabilities of 0.85, 0.10, 0.03 and 0.02 — with k=40 and p=0.9 unchanged:

• Top-k keeps all four.
• Top-p reaches 0.95 at rank 2, so it keeps two tokens.
• The intersection is two tokens.

Same parameters, three different outcomes. That is the practical consequence of pairing a count-based filter with a mass-based one, and it is why "which is more restrictive" has no fixed answer.

Can you set top_p and top_k together? Support by provider

Only one of the three major providers exposes both, and even there it is conditional. Every cell below was checked against the vendor's own documentation on August 25, 2026.

OpenAI. The published OpenAI OpenAPI specification — version 2.3.0, last commit August 24, 2026 — contains zero occurrences of the string top_k. There is no top-k parameter to set. The top_p field is documented as "an alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass," followed by the recommendation to alter "this or temperature but not both." Note the honest limit here: the spec tells you what is exposed, not what OpenAI's serving stack does internally. OpenAI does not publish its internal decoding pipeline, so nobody outside the company can say whether a top-k cut is applied server-side.

Anthropic. The Messages API reference still documents all three fields, and its top_p description is a clean statement of the mechanism: "we compute the cumulative distribution over all the options for each subsequent token in decreasing probability order and cut it off once it reaches a particular probability specified by top_p." Both top_p and top_k carry the note "Recommended for advanced use cases only." All three are now marked deprecated. The migration guide is blunter: "Setting temperature, top_p, or top_k to any non-default value on Claude Opus 4.7 or later models, including Claude Opus 5, returns a 400 error." It adds that the Python SDK v1.0 and later "does not define them, and passing them raises a TypeError," and that "prompting is the recommended way to guide model behavior on Claude Opus 5."

Google. Gemini is the one place both parameters genuinely coexist, and Google documents the order explicitly. From the prompt design strategies guide: "For each token selection step, the topK tokens with the highest probabilities are sampled. Tokens are then further filtered based on topP with the final token selected using temperature sampling." The API discovery document (revision 20260823) adds the constraint: "Gemini models use Top-p (nucleus) sampling or a combination of Top-k and nucleus sampling," and "models running with nucleus sampling don't allow top_k setting." Per-model, an empty top_k attribute "indicates that the model doesn't use top-k sampling, and top_k isn't allowed as a generation parameter." Google also attaches a warning to all three: keep them at defaults on Gemini 3.x models, because changing them "can cause unexpected behavior, such as looping or degraded performance, particularly in complex mathematical or reasoning tasks."

When does the "more restrictive wins" rule break?

There is one situation where the exam answer is not exactly right, and it comes down to the rescaling step.

If an implementation renormalizes the probabilities after the top-k cut and before evaluating top-p, then top-p is no longer measuring the same distribution. The surviving tokens now sum to 1 across a smaller set, so every cumulative total is larger, so the p threshold is reached earlier. The final pool can be strictly smaller than the intersection of the two filters computed independently.

This is not hypothetical. In Hugging Face Transformers — the reference open-source implementation, src/transformers/generation/logits_process.py on main, accessed August 25, 2026 — the processors are appended in the order temperature, top-k, top-p. TopPLogitsWarper then computes sorted_logits.softmax(dim=-1).cumsum(dim=-1) on the current scores, in which the tokens top-k already masked to negative infinity contribute exactly zero. The survivors are renormalized.

Here is a distribution where that matters, with k=3 and p=0.9:

Probabilities:  0.40, 0.40, 0.05, 0.04, 0.04, 0.04, 0.03

Top-k (k=3) alone      → 3 tokens  (0.40, 0.40, 0.05), mass 0.85
Top-p (p=0.9) alone    → 5 tokens  (cumulative: 0.40, 0.80, 0.85, 0.89, 0.93)
Naive intersection     → 3 tokens

Sequential k then p, with renormalization between:
  renormalize the 3 survivors by 0.85 → 0.4706, 0.4706, 0.0588
  cumulative → 0.4706, 0.9412  ← clears 0.9 at the second token
Actual pool            → 2 tokens

Three tokens by the intersection rule, two tokens in the sequential implementation. The rule is a very good approximation and it is the right mental model, but it is an approximation.

You can check the arithmetic yourself:

# Reproduce the edge case. No dependencies beyond the standard library.
probs = [0.40, 0.40, 0.05, 0.04, 0.04, 0.04, 0.03]
k, p = 3, 0.9

def nucleus(dist, p):
    kept, cum = [], 0.0
    for x in sorted(dist, reverse=True):
        kept.append(x); cum += x
        if cum >= p:
            break
    return kept

print("top-p alone:      ", len(nucleus(probs, p)))          # 5

survivors = sorted(probs, reverse=True)[:k]                  # top-k cut
print("top-k alone:      ", len(survivors))                  # 3

mass = sum(survivors)
renormalized = [x / mass for x in survivors]                 # rescale
print("sequential k -> p:", len(nucleus(renormalized, p)))   # 2

Does any of this matter if you do not call an API?

For most readers, no — and that is worth saying plainly rather than burying.

If you use ChatGPT, Claude or Gemini through a browser or a desktop app, you cannot set top_p or top_k. Those interfaces do not expose sampling parameters, and no amount of asking the model in the chat window will change them. The model cannot reach its own decoding configuration.

That leaves three groups of people for whom this genuinely matters:

1. API users on open-weight models you host yourself. Full control, real tuning value, and you can measure the result. This is where sampling parameters earn their keep.
2. API users on hosted frontier models. Increasingly, the answer is "leave them alone." Anthropic rejects non-default values on Opus 4.7 and later. Google strongly recommends defaults on Gemini 3.x. OpenAI does not expose top-k at all and advises against moving temperature and top_p together.
3. People answering an exam question. The expected answer is the intersection rule, and the intersection rule is correct.

Everyone else is better served by the levers they actually control: a clear role, an explicit task, a stated output format, and hard constraints. That is not a consolation prize. Across the three biggest vendors, the documented direction of travel over the past year has been away from exposing decoding knobs and toward prompting as the control surface — Anthropic says so in as many words.

Prompt Architects does not let you set top_p or top_k, and it never will, because the chat interfaces we plug into do not expose them either. What it does is rewrite a loose prompt into a structured one — Role, Task, Format, Constraints, Tone — in under two seconds, which is the lever that is actually available to you in a chat window. If you want the structural patterns without the tool, post 41 covers prompt engineering from first principles and post 49 is the cheat sheet.

The short version

Both filters cut the same sorted list, so the pool is the shorter prefix and the tighter filter governs. Which filter that is depends on the shape of the distribution and changes token by token. OpenAI does not expose top-k at all. Anthropic has deprecated all three parameters on Opus 4.7 and later. Google supports both on some models and applies k before p. And if you are in a chat window rather than an API client, none of it is reachable — write a better prompt instead.