Scoring Against References: Exact, Lexical and Semantic

You will be able to

• Tell exact evaluation from subjective evaluation, and prefer exact methods where a task allows them.
• Use functional correctness, define pass@k, and explain why k must be reported with the score.
• Describe reference-based scoring, its bottleneck in writing and checking references, and the difference from reference-free metrics.
• Apply exact match after normalisation, and recognise where it marks correct answers wrong.
• Compute word overlap, n-gram overlap and edit distance, and name the BLEU/ROUGE family they underlie.
• Explain semantic similarity with embeddings and cosine, and what its reliability and cost depend on.
• Recognise when a similarity metric rewards resemblance over correctness, choose a metric for a task, and say what perplexity can and can't measure.

Topics

1. 1Exact scores and subjective ones
2. 2Does it work? Functional correctness and pass@k
3. 3Reference answers
4. 4Exact match, and where it stops working
5. 5Lexical similarity: do they look alike?
6. 6Semantic similarity: do they mean the same?
7. 7Looking right versus being right
8. 8What perplexity tells you, and wrap-up