Why do language models make things up?

The model is predicting text, not looking anything up

A language model runs one operation over and over: given the text so far, produce a probability distribution over the next small piece of text, then pick one. Everything it can do, from answering to summarizing to writing code, is that loop. A base language model does not by itself verify claims against a record; a surrounding system can add search or retrieval, which still needs reliable sources.

What training leaves behind is a set of weights: a compressed account of how words tend to follow other words across a very large body of text. A fact that appeared often, in consistent form, is encoded strongly and comes back reliably. A fact that appeared once, or in conflicting versions, leaves a faint trace. Nothing in the weights is labeled with a source, and nothing reports that a trace was faint.

Fluent and wrong is the same operation as fluent and right

Producing a true sentence and a false one use the same text-generation process. Pretraining teaches likely text, which can be fluent and wrong. Later training can reward correctness, but errors remain.

This is why the confident tone carries no information. That steady, specific voice is a style learned from confident writing, not a reliable readout of internal certainty. Removing randomness does not fix it either. With sampling set to always take the highest-probability option, a model still returns invented citations, because the invented citation was the highest-probability continuation.

Preference training pushes the same way. An answer is usually rated more useful than an admission of ignorance, so the pressure runs toward saying something rather than declining. A system scored on whether it answers will learn to guess, for the same reason a student guesses on a test that does not penalize wrong answers.

Hallucination is a misleading name for it

The word borrows from perception and suggests a system that normally sees the world and briefly stopped. Nothing is being perceived, and nothing broke. The same mechanism produced the good answer a minute earlier. Calling it an occasional fault sends people looking for a fix that removes it, when the useful framing is a permanent property to design around.

Where it matters most

Risk tracks two things: how rare the required answer is, and how exact it has to be. Summarizing text you supplied is low risk, because the answer is sitting in front of the model. A specific number, name, citation or identifier that has to come out of the weights is high risk, because an exact string must be reconstructed from a thin statistical trace.

Two of these have mechanisms of their own. Events that happened after the training data ends leave no trace to work from, and material that has scrolled out of what the model can hold at once is simply gone from its view. Both are separate questions, linked below.

• Exact strings: citations, case numbers, phone numbers, API signatures, quotes, dates. One answer is right and thousands of plausible near-misses are wrong.
• Long-tail specifics: the minor figure, the small town, the obscure release. Rare in the training text, thinly encoded, still answered in full sentences.
• Anything after the training data ends, where there is no trace to reconstruct from.
• Anything outside the text in front of it, once a conversation runs longer than the model can hold at once.
• Chained arithmetic and counting, where each step is predicted rather than computed.

Can a model be stopped from making things up?

There is no switch that removes only wrong outputs. Grounding answers in supplied evidence can help, but reliability also depends on the model, task, training, and verification.

So the practical move is to change the shape of the task. Give the model the document and ask it to work on that. Ask for a source you can check. Prefer transform-this-text work over recall-this-fact work. Model size can affect performance, but grounding does not guarantee correctness and its effect depends on the task and evidence.