Topic 3

Why models hallucinate

Why does a model state a wrong answer as smoothly as a right one? There is no single cause. Here are the main ones, each visible in GPT-2.

Try this: open the gold prompt and the Bell prompt. In both, the top answer comes from the pattern of the words in front of it, not from chemistry or history.

Five reasons

1. The objective is likely text, not true text. Training rewards predicting the next tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught of real documents (lesson 4). Truth only helps when true statements are what the documents usually say.
2. There is no lookup. Whatever GPT-2 "knows" is spread through its weights, learned from training text. Ji et al. (2022) call this parametric knowledge. Nothing in the forward pass consults a list of facts. Why Sydney beats Canberra is not something this lesson measured. One plausible reason is that Sydney appears near "Australia" in far more web text.
3. Knowledge is frozen at training time. GPT-2's training set, WebText, came from web pages linked from Reddit posts (with Wikipedia pages removed), and the model was released in 2019 (GPT-2 model card). It has learned nothing since. Any model knows only what was in its training data, and Hands-On Large Language Models (Ch. 8, chapter introduction) notes that fluent answers can be wrong or out of date.
4. Patterns beat facts. After "Alexander Graham", Bell has probability 0.999. After "sodium is Na and … gold is", Na wins. AttentionAttentionThe mechanism that lets each token take in information from other tokens, weighted by how relevant they score.Where this is taught is very good at copying from the context (lesson 3), and here it copies the wrong thing.
5. "I don't know" is rarely the likely next text. After "the city of", a city name is what fits grammatically, so GPT-2 names one. Chat models are trained further on answers that people wrote. AI Engineering (Ch. 2, "HallucinationHallucinationFluent, confident output that is false or not supported by the input.Where this is taught") describes a hypothesis from Leo Gao and John Schulman of OpenAI. When those written answers draw on knowledge the model lacks, training on them teaches the model to answer anyway: in effect, to make things up.