Topic 8
What perplexity tells you, and wrap-up
One more number appears in evaluation reports: perplexity. AI Engineering (Ch. 3, "Understanding Language Modeling Metrics", "Perplexity" and "Perplexity Interpretation and Use Cases") defines it from the probabilities a model gives each next token. Low perplexity means the model found a text predictable; high means it was surprised.
The book's rules of thumb: structured text such as HTML gives lower perplexity than everyday prose; a bigger vocabulary gives higher perplexity; and more context gives lower perplexity.
Predict first
Scroll sideways to see the whole diagram
Perplexity scores how predictable a text is to a model, from the probabilities it gives each next token.
Text version
- Perplexity scores how predictable a text is to a model, from the probabilities it gives each next token.
- It's a rough proxy for capability: a model that predicts text badly tends to do badly on tasks too.
- Suspiciously low perplexity on a benchmark suggests the model saw it in training, so its score there means less.
- Very high perplexity flags unusual or garbled text.
- But it doesn't rank chat models: post-training typically raises perplexity even as the model gets better at tasks.
The book lists three uses:
- A rough proxy for capability: a model that predicts text badly tends to do badly on downstream tasks too.
- Detecting contamination: very low perplexity on a benchmark suggests it was in the training data. The same idea helps deduplicate training data.
- Spotting odd text: very high perplexity flags unusual ideas or gibberish.
It also warns that perplexity may not be a good proxy for models that have been post-trained, such as chat models. Post-training with SFT or RLHF teaches a model to do tasks, and usually raises its perplexity. Quantization can change it in unexpected ways too. Module 01 covers the probabilities underneath in probabilities and confidence.
Wrap-up. You can now:
- tell exact scores from subjective ones, and prefer exact where the task allows;
- use functional correctness and pass@k, and report k with it;
- score against references with exact match, word and n-gram overlap, and edit distance;
- explain why semantic similarity needs embeddings, and what it depends on;
- recognise when a metric rewards looking right over being right;
- say what perplexity can and can't tell you.
Quick check