Lesson 7 · 7 topics · 30 min · beginner

Hallucination and the Limits of Next-Token Prediction

You will be able to

Explain why a model trained to predict likely continuations can produce fluent text that is false, and define hallucination (intrinsic and extrinsic).
Read measured GPT-2 next-token distributions for factual prompts and say whether the top answer is right, where the right answer ranks, and why the generated text hides the uncertainty.
Name the main causes of hallucination: the training objective, no built-in fact lookup, knowledge frozen at training time, pattern copying, and rarely saying "I don't know".
Explain how errors snowball during generation because every token is conditioned on the model's own earlier output, and how sampling adds risk.
Treat token probability as a signal to validate, not as the chance of being correct, using measured examples and published calibration results.
Describe mitigations as guidance to test: grounding with retrieval, asking for and checking sources, constrained outputs, abstention, and measuring factual consistency.
Run GPT-2 in the browser and complete the Module 01 capstone: tokens, a greedy loop, a caught hallucination, a temperature change explained from the probabilities, and a three-sentence explanation of the whole path.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 2 "Understanding Foundation Models", section "The Probabilistic Nature of AI", subsection "Hallucination"; Ch. 4 "Evaluate AI Systems", section "Evaluation Criteria", subsection "Generation Capability" ("Factual consistency"); Ch. 6 "RAG and Agents", section "RAG".
Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 6 "Prompt Engineering", section "Instruction-Based Prompting" ("Hallucination"); Ch. 8 "Semantic Search and Retrieval-Augmented Generation", chapter introduction and section "Overview of Semantic Search and RAG".
Survey of Hallucination in Natural Language Generation · Ziwei Ji et al., 2022
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Delong Chen, Wenliang Dai, Ho Shu Chan, Andrea Madotto, Pascale Fung. ACM Computing Surveys (arXiv:2202.03629). Used: Abstract; Section 2 "Definitions" and 2.1 "Categorization"; Section 3.2 "Hallucination from Training and Inference" ("Erroneous decoding", "Exposure Bias", "Parametric knowledge bias"). Open source
Language Models (Mostly) Know What They Know · Saurav Kadavath et al., 2022
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, et al.. arXiv:2207.05221. Used: Abstract. Open source
GPT-4 Technical Report · OpenAI, 2023
OpenAI. arXiv:2303.08774. Used: Section "Limitations", Figure 8 (calibration). Open source
GPT-2 model card (openai-community/gpt2) · Hugging Face / OpenAI
Hugging Face / OpenAI. Used: Sections on limitations and biases, and training data. Open source
Xenova/gpt2 (ONNX weights for Transformers.js) · Xenova (Hugging Face Hub repository)
Xenova (Hugging Face Hub repository). Open source

You will be able to

Explain why a model trained to predict likely continuations can produce fluent text that is false, and define hallucination (intrinsic and extrinsic).

Read measured GPT-2 next-token distributions for factual prompts and say whether the top answer is right, where the right answer ranks, and why the generated text hides the uncertainty.

Name the main causes of hallucination: the training objective, no built-in fact lookup, knowledge frozen at training time, pattern copying, and rarely saying "I don't know".

Explain how errors snowball during generation because every token is conditioned on the model's own earlier output, and how sampling adds risk.

Treat token probability as a signal to validate, not as the chance of being correct, using measured examples and published calibration results.

Describe mitigations as guidance to test: grounding with retrieval, asking for and checking sources, constrained outputs, abstention, and measuring factual consistency.

Run GPT-2 in the browser and complete the Module 01 capstone: tokens, a greedy loop, a caught hallucination, a temperature change explained from the probabilities, and a three-sentence explanation of the whole path.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 2 "Understanding Foundation Models", section "The Probabilistic Nature of AI", subsection "Hallucination"; Ch. 4 "Evaluate AI Systems", section "Evaluation Criteria", subsection "Generation Capability" ("Factual consistency"); Ch. 6 "RAG and Agents", section "RAG".

Hands-On Large Language Models · Jay Alammar et al., 2024

Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 6 "Prompt Engineering", section "Instruction-Based Prompting" ("Hallucination"); Ch. 8 "Semantic Search and Retrieval-Augmented Generation", chapter introduction and section "Overview of Semantic Search and RAG".

Survey of Hallucination in Natural Language Generation · Ziwei Ji et al., 2022

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Delong Chen, Wenliang Dai, Ho Shu Chan, Andrea Madotto, Pascale Fung. ACM Computing Surveys (arXiv:2202.03629). Used: Abstract; Section 2 "Definitions" and 2.1 "Categorization"; Section 3.2 "Hallucination from Training and Inference" ("Erroneous decoding", "Exposure Bias", "Parametric knowledge bias"). Open source

Language Models (Mostly) Know What They Know · Saurav Kadavath et al., 2022

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, et al.. arXiv:2207.05221. Used: Abstract. Open source

GPT-4 Technical Report · OpenAI, 2023

OpenAI. arXiv:2303.08774. Used: Section "Limitations", Figure 8 (calibration). Open source

GPT-2 model card (openai-community/gpt2) · Hugging Face / OpenAI

Hugging Face / OpenAI. Used: Sections on limitations and biases, and training data. Open source

Xenova/gpt2 (ONNX weights for Transformers.js) · Xenova (Hugging Face Hub repository)

Xenova (Hugging Face Hub repository). Open source