Topic 5
Probability is not truth
If the model is unsure, can you just read its probabilities? Lesson 4 ended on this: a probability is not the chance of being right. Here it is again, on the 14 fact prompts.
Predict first
Try this: set the temperature to 0.3 and watch the top bar. about jumps from 0.094 at T = 1 to 0.644 (lesson 4's measurement). The model learned nothing new: the setting sharpened the same distribution. A number shown as "confidence" depends on settings as well as on the model.
What probability does measure
The probability says how typical a token is after this text, in text like the training data. That correlates with being right when the right answer is also the usual one, and fails when a wrong answer is more common (Sydney) or when the prompt sets up a pattern (Na).
Larger, newer models do better, and this has been measured:
- Kadavath et al. (2022), "Language Models (Mostly) Know What They Know", found that larger models are well calibrated on many multiple-choice and true/false questions when the questions are formatted the right way. Models can also be trained to predict whether they know an answer. That generalizes partly across tasks, and its calibration is weaker on new ones.
- The GPT-4 Technical Report (OpenAI, 2023, "Limitations", Figure 8), already cited in lesson 4, found the pre-trained model well calibrated on a subset of a multiple-choice benchmark, and much less so after post-training.
A free-form answer spreads its claim over many tokens, so there is no single token probability to read. Kadavath et al.'s workaround is to ask the model a second question: is the answer it just proposed true? They report encouraging calibration for that probability, improving with model size. So: a low probability is a useful warning, and a high one is evidence, not proof.
Quick check