Topic 6

Reducing hallucination

You cannot train the habit out of a model from your application. You can change what it sees, what it is allowed to say, and what you check afterwards.

Try this: tick Put one true sentence in front of the prompt and compare the gold prompt with and without it. Without it, the context sentence about sodium pulled GPT-2 to the wrong symbol, Na. With a true sentence about gold in front, Au comes out on top, though only at 0.092. Context steers the model either way.

Read that result carefully

It is a toy. The sentence put in front contains the answer word a few tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught before the blank, so the test mostly measures copying, which attentionAttentionThe mechanism that lets each token take in information from other tokens, weighted by how relevant they score.Where this is taught is good at. Real systems have to find the right text first, and they can find the wrong text. Models can also prefer what is in their weights over what is in the prompt (Ji et al., 2022, "parametric knowledge bias").

Four mitigations (guidance, not measurements)

• Ground the answer in retrieved text (Module 05). Retrieval-augmented generation (RAG) fetches documents relevant to the question and puts them in the prompt. AI Engineering (Ch. 6, "RAG") reports that Lewis et al. (2020) found relevant retrieved information reduced hallucinationsHallucinationFluent, confident output that is false or not supported by the input.Where this is taught. Hands-On Large Language Models (Ch. 8, "Overview of Semantic SearchSemantic searchFinding documents by comparing embeddings of the query and the documents, so matches can share meaning without sharing words.Where this is taught and RAG") describes RAG as a way to reduce hallucination and ground a model in a specific dataset, ideally citing its sources.
• Ask for sources and check them. AI Engineering (Ch. 2, "Hallucination") describes verification: ask the model to retrieve the sources each answer rests on. A citation only helps if something, or someone, checks that it exists and says what the answer claims.
• Constrain the output. Short answers leave fewer tokens to make things up (AI Engineering, Ch. 2, "Hallucination"). Answers restricted to the given context, or to a fixed format, are easier to check (structured outputsStructured outputModel output in a fixed machine-readable format, such as JSON with known keys or one label from a list, so code can parse and use it.Where this is taught come in Module 03).
• Allow abstention. Both books describe telling the model to answer only if it knows, and otherwise say "I don't know" (Hands-On Large Language Models, Ch. 6, "Instruction-Based Prompting"; AI Engineering, Ch. 2, "Hallucination"). This only works as far as the model can tell what it knows, which the previous topic showed is partial.

Then measure (Module 04). AI Engineering (Ch. 4, "Evaluation Criteria", "Factual consistency") separates local factual consistency, meaning the answer is supported by the context it was given, from global consistency against open knowledge. Local consistency is much easier to check, which is one more reason to give the model the facts.