Why Does ChatGPT Make Things Up?

TL;DR: ChatGPT makes things up because nothing in its training ever separated knowing from guessing. Pretraining rewards a fluent continuation, post-training rewards a confident answer, and nearly every benchmark scores a wrong answer identically to "I don't know". Under those three incentives, guessing wins. Prompting can shift the balance, but not remove it.

Is ChatGPT lying to you?

No, and the distinction matters more than it sounds. Lying requires knowing the truth and choosing to conceal it. When people ask why does ChatGPT make things up, they usually want a motive, and there isn't one. What there is instead is a chain of ordinary engineering decisions that each, individually, make a confident guess more attractive than an admission of ignorance.

Every article on this topic tells you that language models predict the next token. That is true and almost useless. It explains nothing about why the fabrications are specific, why they are confident, why they cluster on citations and obscure names, or why a bigger model does not simply fix it. This piece goes after them.

Why does next-token prediction produce confident fabrication instead of visible uncertainty?

Because the training objective never presents the model with a labelled false statement to learn from.

The clearest statement of this comes from Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, whose paper was published in Nature on 22 April 2026 as "Evaluating large language models for accuracy incentivizes hallucinations". Their argument runs through a reduction to binary classification: "any language model implicitly answers 'Is this response valid?' for each candidate generation." Generation is therefore at least as hard as classification, and classification errors are well understood.

That reframing produces the paper's sharpest line about what to expect: "Learnable patterns (grammar, spelling, politeness) yield low error rates, whereas non-learnable facts (birthdays, one-off details) produce high hallucination rates in pretrained LLMs."

This is the answer to a question most explainers never ask. Modern models essentially never produce mismatched brackets or misspelled common words, because those follow patterns that survive scale. A stranger's birthday follows no pattern at all. OpenAI's accompanying blog post uses a good analogy: train an image classifier on millions of labelled cat and dog photos and it learns; label each pet photo with the pet's birthday and "this task would always produce errors, no matter how advanced the algorithm."

The paper then quantifies it with what it calls the singleton rate — the fraction of facts that appear exactly once in training. Built on Turing's missing-mass estimator, it yields a lower bound: if 20% of birthday facts appear exactly once in pretraining data, you should expect a base model to hallucinate on at least 20% of birthday questions. Not as a bug. As a floor.

So the fabrication is specific because the model is not producing noise. It is producing the most plausible completion of a pattern, and for arbitrary facts "plausible" and "true" are simply uncorrelated.

Why does the model sound so sure?

Because confidence and correctness are separate signals inside the model, and post-training weakens the connection between them.

The evidence here is OpenAI's own. The GPT-4 technical report reports that "the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced." Figure 8 puts numbers on it: expected calibration error of 0.007 on a subset of MMLU before reinforcement learning, 0.074 after. The caption is blunt: "The post-training hurts calibration significantly."

Read that again, because it inverts the folk theory. The raw statistical model had a usable sense of its own uncertainty. The stage designed to make it a helpful assistant is the stage that flattened it.

Anthropic's interpretability team found the mechanism from the other direction. In "Tracing the thoughts of a large language model" (27 March 2025), they report that in Claude, "refusal to answer is the default behavior: we find a circuit that is 'on' by default and that causes the model to state that it has insufficient information to answer any given question." Asking about a well-known entity activates a competing "known entities" feature that inhibits that default. Hallucination is what happens when the inhibition fires wrongly: "such misfires can occur when Claude recognizes a name but doesn't know anything else about that person... Once the model has decided that it needs to answer the question, it proceeds to confabulate."

That is a mechanistic account of a very familiar experience. You mention a name the model half-recognises, and instead of hesitating it produces a fluent biography of someone who does not exist. The hallucination is not the model failing to notice it doesn't know. It is a recognition signal firing on the shape of a name and switching off the caution that would otherwise have saved you.

Why does it fabricate citations and URLs specifically?

Because a reference is the worst possible object for a next-token predictor: a string of arbitrary values with no internal redundancy, formatted to look authoritative.

The measurement most worth knowing is William Walters and Esther Isabelle Wilder's study in Scientific Reports. They generated 84 short papers on 42 topics and checked all 636 bibliographic citations. Their finding: "55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated. Likewise, 43% of the real GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors."

Note the second half: even the real citations were often wrong in their details. Hyperlinks were "more likely to be found within the fabricated citations" than the real ones, and where links were attached to real works, "about one-third of them are inaccurate."

Now here is the part almost nobody covers. The singleton-rate argument does not explain fabricated references, and the Nature paper says so. Published works get mentioned repeatedly — on a personal page, a CV, a preprint server, a journal index — so their singleton rate should be low, "yet they have been a prominent source of hallucinations." The authors offer a different cause: model misspecification. In their words, "A neural network is less reliable than a database of publication titles, populated based on training data."

That single sentence is the most useful thing you can know about AI citations. The model is not looking anything up. It is reconstructing the shape of a plausible reference from a lossy compression of the literature. Plausible author, plausible journal, plausible volume, plausible DOI. Every component is individually likely and the combination is fictional.

This scales beyond academia. The Tow Center for Digital Journalism at Columbia tested eight AI search products across 200 queries in "AI Search Has a Citation Problem" (March 2025). They found the tools "provided incorrect answers to more than 60 percent of queries" and that "more than half of responses from Gemini and Grok 3 cited fabricated or broken URLs that led to error pages." Their observation on tone is the one to keep: ChatGPT "incorrectly identified 134 articles, but signaled a lack of confidence just fifteen times out of its two hundred responses, and never declined to provide an answer."

If you do research with AI assistance, the companion piece on summarising papers without losing citations covers the workflow side of this.

Why do RLHF and helpfulness training make this worse?

Because the thing being optimised is human approval, and humans approve of confident answers.

Anthropic's "Towards Understanding Sycophancy in Language Models" analysed real human preference data and found that "when a response matches a user's views, it is more likely to be preferred" and that "both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time." Optimising against those preferences "sometimes sacrifices truthfulness in favor of sycophancy."

OpenAI documents the same pressure in reinforcement learning against automated graders. The GPT-5 system card states that models "may learn to be overconfident, cheat, or 'trick' fallible graders, even if their internal reasoning indicates uncertainty, as successful attempts garner a high reward." The internal uncertainty exists. Expressing it scores badly.

And this is where the Nature paper lands its main blow, on evaluation rather than training. "Most leaderboards score each problem... as either correct or incorrect, and abstention is typically graded as incorrect... Under this scoring, abstaining is strictly suboptimal, being penalized as incorrect while an overconfident 'best guess' is optimal for maximizing expected accuracy."

The consequence, in their phrase, is that LLMs are "always in exam mode."

You can see it in a single table from OpenAI's own SimpleQA results:

The older model looks two points better on the only metric most leaderboards print. It is wrong three times as often. OpenAI's comment: "Strategically guessing when uncertain improves accuracy but increases errors and hallucinations."

Why does it get worse with obscure names and long documents?

Two separate effects, often mistaken for one.

Obscurity. Nikhil Kandpal and colleagues showed in "Large Language Models Struggle to Learn Long-Tail Knowledge" (ICML 2023) that a model's ability to answer a factual question "relates to how many documents associated with that question were seen during pre-training", with "strong correlational and causal relationships between accuracy and relevant document count". Their estimate for closing the gap by scale alone is bleak: "today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data." Alex Mallen and colleagues reached the same place from another angle in "When Not to Trust Language Models" (ACL 2023): "LMs struggle with less popular factual knowledge, and... scaling fails to appreciably improve memorization of factual knowledge in the long tail."

So the risky question is not a hard question. It is a question about something rarely written down. A well-known company's founding year is safe. A three-person consultancy's founding year is a coin toss dressed as a fact.

Length. Nelson Liu and colleagues found in "Lost in the Middle" (TACL 2023) that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models." A fact in the middle of a large context window is present but not reliably reachable, and an unreachable fact gets replaced by a plausible one. Anthropic's own guidance implicitly concedes the threshold, recommending quote extraction first "for tasks involving long documents (over 20k tokens)".

If your long chats drift away from what you established early, that is the same effect, and why ChatGPT forgets what you told it covers it in detail.

What does retrieval actually fix, and what does it not?

Retrieval fixes the knowledge gap. It does not fix the incentive to answer.

The evidence for the first half is strong. Mallen and colleagues found "retrieval-augmented LMs largely outperform orders of magnitude larger LMs" on unpopular entities. Kandpal and colleagues concluded retrieval "can reduce the dependence on relevant pre-training information". If you want the fuller comparison, we cover RAG versus fine-tuning versus prompting separately.

The evidence for the second half is equally strong, and it comes from the vendors. OpenAI's GPT-5 system card measures factuality on real ChatGPT production traffic with browsing enabled, and reports that gpt-5-thinking still produced incorrect factual claims 4.5% of the time, with 4.8% of responses containing at least one major factual error. Better than OpenAI o3 at 12.7% and 22.0%. Not zero.

The same document names the specific failure mode that retrieval introduces: in tasks requiring a browsing tool, "previous models would hallucinate information when the tool was unreliable." A broken search does not produce an error message to you. It produces an answer.

Google DeepMind's FACTS Grounding benchmark exists precisely because handing a model a document does not guarantee it stays inside it. Responses are only "judged as factually accurate if they are fully grounded in information contained in the provided document, with no hallucinations" — a bar that models measurably fail to clear, which is why there is a leaderboard rather than a checkbox.

And the Tow Center numbers above are all from products that were searching the live web at the time.

What can you actually do about it in a prompt?

Everything below works on the same principle: make abstention cheap, make claims auditable, or both. Anthropic documents the first three families itself under Reduce hallucinations; the confidence-threshold prompts are modelled on the open-rubric scoring the Nature paper describes, where "if abstaining earns 75% credit, then it is optimal to guess when one is more than 75% likely to be correct."

Force abstention

If you do not know the answer, say "I don't know" and stop. Do not
produce a best guess. An honest "I don't know" is the correct and
preferred answer here, and I will not treat it as a failure.

Score yourself as follows: a correct answer is worth 1 point, "I don't
know" is worth 0.75 points, and a wrong answer is worth 0 points.
Answer only if you are more than 75% confident. Otherwise abstain.

Answer in exactly one of three modes, and label which you used:
[ANSWER] - you are confident and can state the basis.
[PARTIAL] - you can answer part of it; state precisely which part
and what is missing.
[UNKNOWN] - you cannot answer reliably. Say what would let you.

Before answering, ask yourself: do I actually recognise this specific
entity, or only the general shape of the name? If it is only the shape,
say so explicitly and do not generate details.

Demand verifiable citations

For every factual claim, give a source in this format:
CLAIM | SOURCE TITLE | AUTHOR OR PUBLISHER | YEAR | HOW TO VERIFY IT.
If you cannot supply all five fields for a claim, delete the claim
rather than partially citing it.

Do not construct URLs, DOIs, ISBNs or reference numbers. Only reproduce
an identifier if you are reproducing it from a document I have supplied
or a page you have actually retrieved. If you do not have one, write
"no identifier available" instead of generating one.

List your sources first, before writing anything else. Then write the
answer using only those sources. If a point you want to make is not
supported by a listed source, mark it inline as [UNSOURCED].

Ground it in a document you supplied

Answer using ONLY the document below. Do not use your general
knowledge, even where you believe it is correct. If the document does
not answer the question, reply "not covered in the provided document".

Step 1: extract the exact word-for-word quotes from the document that
bear on my question. Do not paraphrase.
Step 2: answer the question using only those quotes.
Step 3: after each sentence of your answer, put the quote number it
rests on.

The document is long. Before answering, list every section heading and
tell me which sections are relevant. Then quote from those sections
only. Do not summarise sections you have not quoted from.

Decompose before you trust

Break your answer into atomic claims, one per line. For each, give a
confidence band: HIGH (widely documented), MEDIUM (I believe this but
would want it checked), LOW (I am reconstructing this). Put the LOW
claims at the top so I check them first.

Separate your response into two sections:
FACTS - things you are asserting as true about the world.
INFERENCE - things you are concluding, estimating or reasoning toward.
Do not blend them.

Run a verification pass

Now review your previous answer as a hostile fact-checker whose job is
to find one fabricated or unsupported claim in it. For each claim,
either quote the supporting evidence or mark it RETRACTED. Output only
the retractions and the corrected answer.

Answer this question twice, independently, without looking at your first
attempt. Then compare the two. Report only what both agree on. Where
they disagree, say "inconsistent - do not rely on this".

For the answer you just gave, write the strongest case that it is
wrong. Then tell me which of the two positions the available evidence
actually supports, and how confident you are.

Handle premises and stakes

Before answering, check my question for false premises. If it assumes
something that is not true - a product that does not exist, an event
that did not happen, a feature that was never shipped - say so instead
of answering the question as asked.

This is going into [a legal filing / a medical summary / production
code]. A wrong answer costs far more than a missing one. Flag every
claim you would not personally stake a professional reputation on, and
prefer "I don't know" over any answer you would need to hedge.

Give me your answer, then a separate line: "What would change my mind:"
listing the specific evidence that would show this answer to be wrong.
If you cannot name any such evidence, say why not.

The self-consistency prompt above is worth singling out, because it is close to the mitigation the Nature authors tested: generate two independent responses, have the model judge whether they agree, and abstain when they do not. Their honest finding was that "the mitigation cuts errors but also reduces accuracy across models" — you lose some right answers along with the wrong ones. Whether that trade is worth it depends entirely on what a wrong answer costs you. For a first draft, probably not. For anything a client sees, obviously yes.

What none of this fixes

None of these prompts give the model knowledge it never had. They change what it does in the absence of knowledge. That is a real change, and a smaller one than most tools imply.

Three honest limits are worth stating plainly. First, the effect is probabilistic — a prompt that tells the model abstention is acceptable raises the abstention rate, it does not guarantee abstention on the specific question you care about. Second, confidence self-reports are themselves model outputs, subject to the same calibration damage documented in the GPT-4 report, so a stated "HIGH confidence" is evidence rather than proof. Third, pushing back can make things worse: given the sycophancy findings, "are you sure?" is a weaker instrument than "re-derive this from the quotes you cited."

OpenAI's own Model Spec publishes the ranking every user should internalise: "confident right answer > hedged right answer > no answer > hedged wrong answer > confident wrong answer." The whole practice of prompting against hallucination is moving outputs leftward along that line.

The direction of travel is genuinely good, and slower than the marketing suggests. OpenAI's GPT-5.5 system card, measured on conversations users had flagged as containing factual errors, reports that "GPT-5.5's individual claims are 23% more likely to be factually correct, and its responses contain a factual error 3% less often" than GPT-5.4. Meaningful. Not solved. The Nature authors' conclusion is the right one to end on: "further progress may depend as much on what we measure as on what we build."

Until the scoreboards change, the model in front of you is still taking an exam. Write your prompts as though you are the invigilator who has told it, out loud, that a blank answer beats a wrong one. And if you are chasing a related symptom, reflexive apologising and answers that stop mid-sentence have entirely different causes than this one.