Topic 8

Finding the failed step, and module wrap-up

When a RAG answer is wrong, the error analysis loop from Module 04 applies, with a checklist specific to retrieval. Module 04's own example found “retrieval missed the right page” as the most common failure. Walk each wrong answer back through the pipeline until you find the first step that failed.

Predict first

A user asks about the refund window and gets last year's 14-day policy. The current page says 30 days. Logs show the retrieved chunk was the old policy page, ranked first. Which step failed first?

Ingestion: the old page is still indexed, so retrieval correctly found what it was givenThe generatorThe rerankerChunking

Where did this RAG answer go wrong?Step 1 / 5

Scroll sideways to see the whole diagram

First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.

Text version

First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
Is it in a chunk that says what it's about? If it was split or lost its context, fix chunking (lesson 4).
Did retrieval return the chunk at all? If not, look at the retrievers and the query (lessons 2–4).
Was it returned but cut: ranked below k, or over the token budget? Look at reranking and the pool size (lesson 3).
Only if it was in the prompt is it a generation problem. Logging every step's input and output is what makes these questions answerable.

Every question in that tree needs logs to answer: the rewritten query, the retrieved chunks with scores, what the reranker kept, what went into the prompt, and the answer. Without them, every failure looks like “the model got it wrong”.

This lesson closes the module. Each lesson took one part of a RAG system:

How RAG works: retrieve, then generate; when to retrieve at all; why retrieval caps the answer.
Term-based and embedding retrieval: BM25 term by term, k-NN and ANN, and their opposite strengths.
Hybrid search and reranking: fusing rankings, when fusion hurts, and cross-encoders on a shortlist.
Chunking and indexing: boundaries, overlap, structure, size, and giving chunks their context.
Evaluating RAG: retrieval metrics, citations, faithfulness, abstention, and finding the failed step.

Wrap-up. From this lesson, you can now:

evaluate a RAG system component by component and end to end;
compute context precision and recall, and recall@k at the cut you actually use;
compute reciprocal rank, average precision and MAP, and say what nDCG adds;
sort results into retrieved/missed × right/wrong, and know where each points;
measure citation recall, citation precision and faithfulness, and why they differ;
design for abstention, and measure refusals on both answerable and unanswerable questions;
trace a wrong answer back to the first step that failed.

Quick check

In one sentence, what does this module say about building RAG?

Use the biggest modelAlways use hybrid search with a rerankerChunk at 512 tokensRetrieval decides what the generator can know, so build each stage for its job and measure each stage separately, as well as the final answer

Finding the failed step, and module wrap-up

Predict first

A user asks about the refund window and gets last year's 14-day policy. The current page says 30 days. Logs show the retrieved chunk was the old policy page, ranked first. Which step failed first?

Ingestion: the old page is still indexed, so retrieval correctly found what it was givenThe generatorThe rerankerChunking

Where did this RAG answer go wrong?Step 1 / 5

Scroll sideways to see the whole diagram

First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.

Text version

First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
Is it in a chunk that says what it's about? If it was split or lost its context, fix chunking (lesson 4).
Did retrieval return the chunk at all? If not, look at the retrievers and the query (lessons 2–4).
Was it returned but cut: ranked below k, or over the token budget? Look at reranking and the pool size (lesson 3).
Only if it was in the prompt is it a generation problem. Logging every step's input and output is what makes these questions answerable.

This lesson closes the module. Each lesson took one part of a RAG system:

How RAG works: retrieve, then generate; when to retrieve at all; why retrieval caps the answer.

Term-based and embedding retrieval: BM25 term by term, k-NN and ANN, and their opposite strengths.

Hybrid search and reranking: fusing rankings, when fusion hurts, and cross-encoders on a shortlist.

Chunking and indexing: boundaries, overlap, structure, size, and giving chunks their context.

Evaluating RAG: retrieval metrics, citations, faithfulness, abstention, and finding the failed step.

Wrap-up. From this lesson, you can now:

evaluate a RAG system component by component and end to end;

compute context precision and recall, and recall@k at the cut you actually use;

compute reciprocal rank, average precision and MAP, and say what nDCG adds;

sort results into retrieved/missed × right/wrong, and know where each points;

measure citation recall, citation precision and faithfulness, and why they differ;

design for abstention, and measure refusals on both answerable and unanswerable questions;

trace a wrong answer back to the first step that failed.

Quick check

In one sentence, what does this module say about building RAG?