What Is LLM Hallucination?

An LLM hallucination is an answer from a language model that sounds right and is not. The model states a made-up fact, citation, number, or policy with the same confidence as a true one.

For a chat toy, that is a nuisance. For a production agent, a hallucinated order number or contract clause can trigger a real action.

Why models do it

A language model is trained to predict likely next words. It is not a database that looks things up.

OpenAI’s September 2025 research, Why language models hallucinate, gives two causes. First, pretraining has no true/false labels. Patterns like spelling and grammar are learnable. Arbitrary low-frequency facts, like a person’s birthday, are not predictable from patterns, so the model guesses. Second, evaluation methods reward guessing. Most benchmarks grade on accuracy alone, so a model that guesses scores better than one that admits uncertainty. OpenAI’s proposed fix is to penalize confident errors more than uncertainty and give partial credit for appropriate “I don’t know” answers.

Earlier work covered the same ground from a research angle. A 2022 survey of hallucination in natural language generation by Ji and colleagues reviewed metrics, mitigation methods, and task-specific problems in summarization, dialogue, question answering, and more.

The practical takeaway: hallucination is a property of how these models work. You can reduce it. You cannot assume it away.

Where it shows up in agents

Invented facts. Prices, dates, policy terms, names.
Invented sources. A citation that looks real and is not.
Invented tool arguments. An ID that was never in the data.
False completion. “I sent the email” when no tool call ran.
Stale knowledge. A confident answer about something that changed after training.

Agents make it costlier because the false output can feed the next step. A wrong customer ID in step two corrupts steps three through six.

What reduces it

Give the model the source. Retrieval-augmented generation puts relevant documents in the prompt. The original RAG paper framed it as a way to give models explicit non-parametric memory, and noted that provenance and updating knowledge were open problems for models relying on stored parameters alone. See RAG system.
Let it say no. Instruct the model to answer only from the provided context and to say “not found” otherwise. Then test that it does.
Constrain outputs. Use structured outputs and enumerated values. A field limited to five valid statuses cannot hold a sixth invented one.
Verify in code. Check that an order ID exists before acting on it. Check that a quoted passage appears in the source.
Show sources to the user. Links to the retrieved passages let a person check in seconds.
Evaluate for it. Build test cases where the right answer is “I don’t know” and measure how often the agent gets that right. See agent evaluation and LLM as a judge.

A short example

Say an HR assistant answers questions about leave policy. Without retrieval, it may invent a carryover rule. With retrieval, the policy PDF is in context and the assistant must quote the section it used. A code check confirms the quoted text exists in the PDF. If retrieval returns nothing relevant, the assistant says it cannot find the policy and routes the question to HR. The failure moves from “confident wrong answer” to “visible gap.”

How to budget for it

Zero is not a realistic target. Set an acceptable error rate per task type, based on the cost of a mistake. A draft internal summary can tolerate more. A payment instruction cannot. Pair higher-risk outputs with checks or a human reviewer.

Common mistakes

Treating RAG as a cure. Bad retrieval gives the model bad context, and it will still answer.
Testing only questions the documents can answer.
Measuring only accuracy and ignoring how often the agent should have abstained.
Letting the model produce IDs, amounts, or dates that code could look up.
Assuming the next model release fixes it.

How to measure it

Pick a sample of real questions and have a person mark each answer as supported, unsupported, or wrong. Track the unsupported rate over time and by question type. Include questions the sources cannot answer, because the right behavior there is to abstain.

Break results down by cause. Did retrieval miss the document? Did the model ignore the passage? Did it invent a field the tool never returned? Each cause has a different fix, and a single overall accuracy number hides which one you have.

Grounding is the main fix. RAG system is the usual way to implement it. Semantic search and vector databases power the retrieval step. Context limits matter too, so see LLM context window.