Lesson 5 · 8 topics · 24 min · intermediate

Evaluating RAG: Retrieval, Grounding and Failures

You will be able to

Evaluate a RAG system component by component (embeddings, retrieval, generation) and end to end.
Compute context precision and recall, and recall@k at the cut the generator actually receives.
Compute reciprocal rank, average precision and MAP, and explain what nDCG adds.
Sort test results into retrieved or missed × right or wrong, and say where each cell points.
Measure citation recall, citation precision and faithfulness, and explain how they differ.
Design for abstention, and measure refusals on both answerable and unanswerable questions.
Trace a wrong RAG answer back to the first pipeline step that failed.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 6 "RAG and Agents": "Retrieval Algorithms" > "Comparing retrieval algorithms".
Hands-On Large Language Models · Jay Alammar & Maarten Grootendorst, 2024
Jay Alammar & Maarten Grootendorst. O'Reilly. Used: Ch. 8 "Semantic Search and Retrieval-Augmented Generation": "Caveats of dense retrieval"; "Retrieval Evaluation Metrics"; "From Search to RAG"; "RAG Evaluation".
Evaluating Verifiability in Generative Search Engines · Nelson F. Liu et al., 2023
Nelson F. Liu, Tianyi Zhang & Percy Liang. arXiv:2304.09848 (Findings of EMNLP 2023). Used: Abstract (fetched 2026-10-23). Open source
Faithfulness (Ragas documentation) · Ragas
Ragas. Used: Definition and formula (fetched 2026-10-23). Open source

You will be able to

Evaluate a RAG system component by component (embeddings, retrieval, generation) and end to end.

Compute context precision and recall, and recall@k at the cut the generator actually receives.

Compute reciprocal rank, average precision and MAP, and explain what nDCG adds.

Sort test results into retrieved or missed × right or wrong, and say where each cell points.

Measure citation recall, citation precision and faithfulness, and explain how they differ.

Design for abstention, and measure refusals on both answerable and unanswerable questions.

Trace a wrong RAG answer back to the first pipeline step that failed.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 6 "RAG and Agents": "Retrieval Algorithms" > "Comparing retrieval algorithms".

Hands-On Large Language Models · Jay Alammar & Maarten Grootendorst, 2024

Jay Alammar & Maarten Grootendorst. O'Reilly. Used: Ch. 8 "Semantic Search and Retrieval-Augmented Generation": "Caveats of dense retrieval"; "Retrieval Evaluation Metrics"; "From Search to RAG"; "RAG Evaluation".

Evaluating Verifiability in Generative Search Engines · Nelson F. Liu et al., 2023

Nelson F. Liu, Tianyi Zhang & Percy Liang. arXiv:2304.09848 (Findings of EMNLP 2023). Used: Abstract (fetched 2026-10-23). Open source

Faithfulness (Ragas documentation) · Ragas

Ragas. Used: Definition and formula (fetched 2026-10-23). Open source