Topic 5

Lexical similarity: do they look alike?

When exact matchExact matchA scorer that passes an output only if it equals the expected answer exactly. Suits short, exact answers; fails correct answers that are formatted differently.Where this is taught is too strict, you can measure how much the output looks like the reference. AI Engineering (Ch. 3, "Similarity Measurements Against Reference Data", "Lexical similarity") describes three common ways:

• Word overlap: split both texts into tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught, here words, and count how many of the reference's words appear in the output.
• n-gram overlap: the same with runs of n words. Pairs (bigrams) start to reward word order.
• Edit distance, also called fuzzy matching: the number of single-character edits needed to turn one text into the other, counting insertions, deletions and substitutions. "bad" is one edit from "bard" and three from "cash". Some matchers also count swapping two letters as one edit.

Named metrics such as BLEU, ROUGE, METEOR++, TER and CIDEr are variations on this idea, differing in exactly how they count overlap. The book notes they were common before foundation models, especially for translation, and that fewer benchmarks use them now. WMT, COCO Captions and GEMv2 are among those that still do.