Topic 4

Errors that snowball

Generation is a loop: each picked tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught becomes input for the next step (lesson 4). That loop has a consequence for errors.

Try this: compare the greedy run with the sampled runs for “The largest planet in our solar system is”. Greedy puts the planet “about 1.5 billion light years away” (lesson 4's measurement), and the next sentence repeats that figure.

The model believes itself

The model treats its own output exactly like your prompt: it is all just context. AI Engineering (Ch. 2, "HallucinationHallucinationFluent, confident output that is false or not supported by the input.Where this is taught") describes two related ideas:

• Self-delusion (Ortega et al., DeepMind, 2021): the model does not separate the text it was given from the text it generated. Once it has written a slightly wrong claim, it expands on it.
• Snowballing (Zhang et al., 2023): after an early wrong assumption, a model keeps going to justify it, and can then fail questions it would otherwise get right.

Ji et al. (2022, section "Hallucination from Training and Inference") name a related training-time cause, exposure bias. During training the model always predicts from real text. During generation it predicts from its own earlier output, and errors can build up as the text gets longer.

A right first token does not protect the rest of the continuation. In GPT-2's greedy runs, each of these began with the correct answer:

• Everest: “about 1,000 feet above sea level”. Its summit is about 29,031 feet (8,848.86 m).
• Romeo and Juliet: “based on the play "The Tempest,"”. The story comes from an Italian tale by Matteo Bandello, which Arthur Brooke turned into a verse poem and William Painter retold in prose.
• Pride and Prejudice: “a collection of essays by Austen”. It is a novel.
• Mercury: “about 1.5 times the distance from the Sun to the Sun”, which does not mean anything.

Sampling adds risk

Sampling can pick a token the model itself rated unlikely, and every later step then has to continue from it. Ji et al. cite Dziri et al., who found that a decoding strategy that adds diversity, such as top-kTop-kSampling only among the k most likely tokens, after renormalising their probabilities.Where this is taught sampling, goes with more hallucination. Ji et al. suggest the added randomness is the reason, but that is their conjecture, not a measured cause.