Topic 6
Semantic similarity: do they mean the same?
Lexical metrics measure whether texts look alike, not whether they mean the same thing. AI Engineering (Ch. 3, "Similarity Measurements Against Reference Data", "Semantic similarity") gives the fix: turn each text into an embedding, a vector meant to capture its meaning, and compare the vectors, usually with cosine similarity. Identical directions score 1; opposite ones score −1. Module 01 builds this up in measuring similarity.
Predict first
Scroll sideways to see the whole diagram
Two sentences with few words in common.
Text version
- Two sentences with few words in common.
- They mean the same thing. Word overlap scores them as far apart; a meaning-based score shouldn't.
- Two sentences made of exactly the same words.
- Word overlap gives them 100%. Pairs of adjacent words catch some of it; only meaning catches all of it.
Metrics in this family include BERTScore, which uses BERT's embeddings, and MoverScore. The book weighs them up:
- Fewer references needed: a correct rewording can score high without matching any reference's words.
- Only as good as the embedding model: two texts with the same meaning can still score low if the embeddings are poor. So the book notes semantic similarity is arguably subjective, since different embedding models give different scores, even though each score is computed exactly.
- Compute cost: you have to run an embedding model on every output and reference.
Quick check