Topic 4
Exact match, and where it stops working
The simplest comparison: does the output match a reference exactly? AI Engineering (Ch. 3, "Similarity Measurements Against Reference Data", "Exact match") says this works for tasks with short, exact answers: simple arithmetic, trivia, an account balance, filling in a blank.
Scorers usually normalise first, such as trimming spaces and ignoring case, and they accept any of several references. A looser variant accepts any output that contains the reference. Module 03's scorers topic shows how that can pass a wrong answer.
Predict first
“How much is the student discount?”
Illustrative answers · metrics computed in your browser
| Answer | Right? | Exact | Words | Pairs | Edit |
|---|---|---|---|---|---|
| “Students get 20% off with a school email address.”Word for word the reference. | right | yes | 100.0% | 100.0% | 100.0% |
| “With a school email, you get a 20% discount.”Right, in different words. | right | no | 66.7% | 37.5% | 14.3% |
| “20%.”Right, and as short as it gets. | right | no | 11.1% | 0.0% | 8.2% |
| “Students get 50% off with a school email address.”One number changed: wrong. | wrong | no | 88.9% | 75.0% | 98.0% |
| “Yes, students get a discount with a school email address.”Doesn't say how much. | wrong | no | 77.8% | 62.5% | 75.4% |
That's the limit the book points to: beyond short answers, exact match rarely works. A sentence can be translated many ways, and the longer the text, the more right answers there are. You can't write them all down, so a correct answer gets marked wrong.
Look at the other columns too. The wrong answer, which changes only the number to 50%, scores 98% on edit similarity, while a correct rewording scores 14%. The next topics explain why.
Quick check