Topic 2

Context precision and recall

AI Engineering (Ch. 6, "Comparing retrieval algorithms") names two retrieval metrics that RAG evaluation frameworks often use:

Context precision (also called context relevance): of all the documents retrieved, what share is relevant to the query?
Context recall: of all the documents relevant to the query, what share was retrieved?

To compute them you need an evaluation set: test queries, a set of documents, and for each query a label on each document saying relevant or not. People or AI judges can do the labelling.

Predict first

Which is harder to compute in production, context precision or context recall?

Precision: you need every document labelledBoth are equally easyNeither can be computedRecall: you need to know every relevant document in the whole collection, so every document must be judged for each query

The help-centre queries make this easy: each has exactly one relevant article, labelled by hand. With one relevant document, recall@k is just “was it in the top k?”, averaged over queries. Here are the five strategies from lessons 2 and 3:

Scoring the retrievers

Six help-centre queries, one relevant article each · rankings from lessons 2 and 3

Strategy	Ranks	Recall@1	Recall@3	MRR
BM25	2 1 1 – 2 –	33%	67%	0.50
Embeddings	1 1 1 1 2 1	83%	100%	0.92
BM25, then embeddings	1 1 1 – 2 –	50%	67%	0.58
Fusion (RRF)	1 1 1 2 2 2	50%	100%	0.75
Fusion + rerank	1 1 1 1 2 1	83%	100%	0.92

Recall@k: share of queries whose relevant article is in the top k. MRR: mean of 1 / rank of the relevant article (0 if it wasn't returned).

Read the columns together. Fusion has the same recall@3 as embeddings (every answer is in its top 3) but much lower recall@1. If your generator gets three passages, fusion and embeddings look the same; if it gets one, they don't. Choose k first, then measure recall at that k.

One more: the last column, MRR, rewards putting the relevant document high. That's the next page.

Quick check

You'll send 3 passages to the generator. Which metric best matches what you need from retrieval?

Recall@1Recall@3: whether the relevant passage is anywhere in what the generator will seeRecall@100Precision@100