Topic 3

Metrics that care about order

Precision and recall ignore order: a relevant document at rank 1 counts the same as one at rank 3. When order matters, AI Engineering (Ch. 6, "Comparing retrieval algorithms") lists three rank-aware metrics: MRR (mean reciprocal rank), MAP (mean average precision) and nDCG (normalised discounted cumulative gain).

• Reciprocal rank is 1 / the rank of the first relevant result: 1 at rank 1, 0.5 at rank 2, 0 if it's missing. MRR averages it over queries. On the previous page, BM25BM25A keyword-search scoring function that ranks documents by the query words they contain, weighting rare words more and long documents less.Where this is taught scored 0.50 and embeddingsEmbeddingA vector of numbers that represents a token or a text, arranged so that similar meanings end up close together.Where this is taught 0.92.
• Average precision handles queries with several relevant documents.

Hands-On Large Language Models (Ch. 8, "Retrieval Evaluation Metrics") builds average precision up from a small example. Two systems each return three results with one relevant document among them: system 1 has it first, system 2 has it third.

The lab starts with relevant results at positions 1 and 3: (1/1 + 2/3) / 2 ≈ 0.83. Mark more results relevant, or move them, and watch each precision@k. Then add a relevant document the system never retrieved: it adds nothing to the sum but still counts in the denominator, so AP drops. That's how AP penalises missing documents as well as low rankings. MAP is the mean of AP over all test queries.

nDCG goes one step further: it allows graded relevance (this document is very relevant, that one only somewhat) and discounts gains further down the list. The book recommends the "Evaluation in information retrieval" chapter of Manning, Raghavan and Schütze's Introduction to Information Retrieval for more.