Topic 3
Reference answers
When you can't run the output, the next option is to compare it with known good answers. AI Engineering (Ch. 3, "Exact Evaluation", "Similarity Measurements Against Reference Data") describes the setup: each test example is an input plus one or more reference responses, also called ground truths. A French sentence might have several acceptable English translations. Metrics that need references are reference-based; those that don't are reference-free.
Scroll sideways to see the whole diagram
Each test input comes with one or more reference answers, also called ground truths.
Text version
- Each test input comes with one or more reference answers, also called ground truths.
- The system produces its own output.
- The output is scored by how similar it is to the closest reference. A right answer that matches none of them can still score low.
- References cost time and money to write. Generating them with a model is cheaper, but they still need a person to check them.
Predict first
The book notes that references written by people treat human performance as the gold standard, and are slow and expensive to produce. More teams generate references with a model instead. Those still need a person to review them, but reviewing takes far less work than writing from scratch.
The output can then be compared with the references in four ways: ask an evaluator whether two texts are the same, check for an exact match, measure lexical similarity (how alike they look), or measure semantic similarity (how alike they mean). The next three topics take the hand-written metrics one at a time.
Quick check