Logprobs: Reading the Model's Confidence

TL;DR: Logprobs explained simply: they're the log-probability the model assigned to each token it output, returned alongside the response so you can see how confident the model's own decoding process was. OpenAI, Google Gemini and Cohere expose them today; Anthropic's Claude API does not, on any endpoint. A high logprob means the token was a fluent continuation, not that the claim it's part of is true.

What are logprobs, exactly?

At every position in a generated response, the model doesn't pick one token out of nowhere. It computes a probability distribution over its entire vocabulary (every possible next token, each with a likelihood) and then samples (or greedily picks) one. A logprob is the natural logarithm of the probability assigned to the token that was actually chosen. Because probabilities are fractions between 0 and 1, their logs are negative numbers close to zero for likely tokens (-0.02 is roughly 98% likely) and large negative numbers for surprising ones (-5 is under 1% likely). Logs are used instead of raw probabilities because multiplying many small fractions together underflows; adding their logs doesn't.

When an API returns logprobs, it is exposing that internal number for each output token, and optionally the logprobs of the runner-up tokens the model considered instead. That second part, the alternatives, is where providers diverge sharply, which is the part most explainers skip.

Why doesn't a high logprob mean the answer is true?

Because the number measures something narrower than truth: how unsurprising this token was, given everything the model has generated so far, including any wrong turn it already committed to.

Once a model has started a sentence with a fabricated name or a wrong figure, every token that grammatically completes that sentence gets a high logprob: the model is effectively being asked what word comes next given this beginning, not whether the beginning was ever correct. OpenAI's own GPT-4 technical report shows this gap directly, and it's worth reading exactly what it measured rather than the folk version of it. The paper's Figure 8 caption describes a calibration test run "on a subset of the MMLU dataset," where the x-axis bins are "according to the model’s confidence (logprob) in each of the A/B/C/D choices for each question" and the y-axis is the actual accuracy within each bin. On the pre-trained base model, the paper states plainly that it "is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct)." Then comes the finding that matters here: "after the post-training process, the calibration is reduced," and the same figure's caption states it without hedging: "The post-training hurts calibration significantly."

So a -0.01 logprob on a wrong answer isn't a contradiction. It's the expected behavior of a system whose training rewarded confident, fluent answers over hedged or abstaining ones, running on a metric that only ever measured fluency in the first place.

How do you actually request logprobs from OpenAI?

OpenAI supports logprobs on both of its current APIs, with slightly different plumbing on each. On Chat Completions, set logprobs to true and, optionally, top_logprobs to an integer between 0 and 20 to also get the runner-up tokens at each position. OpenAI's own API reference states the constraint directly: top_logprobs must have logprobs set to true if it's used.

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-6-astra",
    "messages": [{"role": "user", "content": "What year was the Eiffel Tower completed?"}],
    "logprobs": true,
    "top_logprobs": 3
  }'

The response attaches a logprobs object to each choice, with one entry per output token carrying that token's own logprob plus a top_logprobs array of the alternatives the model didn't pick, each with its own logprob. On the newer Responses API, the boolean flag is gone; you instead ask for it via the include parameter ("message.output_text.logprobs") alongside the same top_logprobs integer, capped at 20 either way.

Does every provider expose them the same way?

No, and the gaps are not cosmetic: some providers return a genuinely different shape of information, and one provider has been actively removing support from its newest models.

Google Gemini mirrors OpenAI's shape closely on its standard generateContent endpoint: set responseLogprobs to true and logprobs to an integer up to 20, and the response candidate carries a logprobsResult object with topCandidates and chosenCandidates arrays. One carve-out worth knowing if you're building a voice or live app: Gemini's own reference explicitly lists responseLogprobs and logprobs among the GenerationConfig fields the Live/BidiGenerateContent streaming setup does not support. Standard, non-streaming calls are unaffected.

Cohere's Chat API returns logprobs too, but in a narrower shape worth knowing before you build around it: its documented response includes a logprobs list of LogprobItem objects, each carrying token_ids, the text chunk, and the log probability of the tokens actually used to construct it. There is no equivalent of top_logprobs in that schema: you get the probability of what the model said, not a ranked list of what it didn't say. If your use case depends on comparing the winning token against its runners-up, Cohere's current API can't give you that; OpenAI's and Gemini's can.

Why does Anthropic's Claude API return nothing here?

This is the one gap builders hit most often, usually mid-migration. Anthropic's Messages API reference has no logprobs field anywhere in its parameter or response schema: not hidden, not beta-gated, simply absent. The clearest confirmation comes from Anthropic's own OpenAI-SDK compatibility layer, which exists specifically to map OpenAI-shaped requests onto Claude models and documents, field by field, what happens to parameters Claude doesn't support. It lists logprobs and top_logprobs both as "Ignored": sent in the request, accepted without an error, and silently dropped. If your code was written against OpenAI's shape and you point it at Claude through that compatibility layer, it won't fail. It will just stop giving you the numbers you were reading.

Anthropic hasn't published a stated reason for the omission, and this piece won't invent one. What's verifiable is only the outcome: if your evaluation, confidence-gating or classification pipeline depends on token-level probabilities, Claude models are not a drop-in option for that specific piece, regardless of how well they perform on the underlying task.

What happened to xAI's Grok logprobs?

xAI is the clearest case of a provider that has actively stopped exposing logprobs, and it's recent enough that older explainers of this topic don't cover it. xAI's own API reference documents logprobs as a boolean field with this exact caveat in its description: "Not supported by models grok-4.20 and newer; the field will be silently ignored if set." The paired top_logprobs parameter carries the same restriction. Older Grok models in the same family document the parameter normally: "An integer between 0 and 8 specifying the number of most likely tokens to return at each token position." That's not an API that never had the feature. It's one where the newest flagship models dropped it, silently, the same way Anthropic's compatibility shim does.

That "silently ignored" phrasing is worth sitting with. Nothing in either provider's error handling tells your code the number it's reading is now always the same default value. If a script was built assuming logprobs would always be there, it keeps running and keeps producing output; the confidence signal just quietly stops meaning anything.

What can you actually do with logprobs in practice?

The single most useful pattern is confidence gating on tasks where the model outputs one discrete token that decides something: a classification label, a yes/no, a routing decision. Force the answer into that shape, read the logprob on that one token, and use it as a cheap, numeric proxy for "should a human check this," instead of asking the model to grade its own confidence in words, which runs into the exact calibration problem above.

import math

def confidence_from_logprob(logprob: float) -> float:
    """Convert a returned logprob back to a 0-1 probability."""
    return math.exp(logprob)

# Example: OpenAI chat completion response with logprobs=true
top_token = response.choices[0].logprobs.content[0]
p = confidence_from_logprob(top_token.logprob)

if p < 0.70:
    route_to_human_review(response)
else:
    accept_automatically(response)

This works because you're reading a number the model already computed, rather than asking it a second question ("how confident are you?") whose answer is itself just more generated text, subject to the same post-training miscalibration as everything else it says. It's also cheap: no second API call, no extra latency beyond the marginal cost of returning a field that was already computed internally.

A second pattern worth knowing: summing or averaging the logprobs across a whole generated span gives you a rough proxy for how "surprised" the model was by its own output overall: a crude, cheaper cousin of perplexity. It's useful for flagging an unusually low-confidence generation for review at the response level, not for judging any single factual claim inside it.

What logprobs can't tell you

Three limits worth stating plainly, because tooling built on logprobs tends to oversell past them. First, a logprob reflects the model's internal distribution at generation time: it says nothing about whether the retrieved document you gave it actually supports the claim, or whether the claim is grounded in anything real at all when no document was supplied. Second, it's per-token, not per-claim: a fabricated name spelled in common, high-frequency word-parts can score a perfectly ordinary logprob even though the entity doesn't exist. Third, changing temperature or top_p changes the distribution the token was actually sampled from, so a logprob from one call at temperature: 1.0 isn't directly comparable to one from another call at temperature: 0.2: you're reading two different distributions, not the same ruler twice.

If what you actually need is to reduce how often the model states something ungrounded in the first place, rather than measure its confidence after the fact, that's a prompting problem, not a parameter, and a different piece on this site covers it directly.

Logprobs are a genuinely useful instrument once you know what they're an instrument for: reading how sure the decoding process was, one token at a time, from a provider that still returns the number. That is smaller, more mechanical and less reassuring than "confidence score" makes it sound, and if you're on Claude or a current Grok model, it isn't available at all. If your API costs are climbing while you build this kind of pipeline, why your API bill is so high is worth a read before you add another field to every call, and if you're pulling logprobs out of a streamed response specifically, streaming responses and when not to covers the parts of that that behave differently from a single blocking call. For a wider view of what else differs by provider on the same request, the token-level defaults are covered in the LLM parameter cheat sheet.