Topic 1

Measure the parts, and the whole

A RAG system can be wrong in more places than a plain model call: the document, the chunk, the retriever, the reranker, the cut into the prompt, or the generator. A single end-to-end score says that something went wrong. It can't say where.

AI Engineering (Ch. 6, "Comparing retrieval algorithms") sums up how to evaluate a RAG system: component by component and end to end. Evaluate the retrieval quality, the final outputs and, for embedding-based retrieval, the embeddings.

Predict first

Your RAG assistant's accuracy drops from 82% to 74% after a release that changed both the chunk size and the prompt. What do you need to find out which change caused it?

A retrieval metric on the same questions: if retrieval held steady, the prompt change is the suspect; if it fell, look at chunkingOnly the end-to-end accuracyA bigger test setA new judge model

What to evaluate in a RAG systemStep 1 / 4

Scroll sideways to see the whole diagram

For embedding-based retrieval, the embeddings themselves can be evaluated: good if more-similar documents get closer vectors.

Text version

For embedding-based retrieval, the embeddings themselves can be evaluated: good if more-similar documents get closer vectors.
Retrieval is evaluated on whether the relevant documents come back, and in what order.
Generation is evaluated on whether the answer follows from what was retrieved.
And the whole system on whether the answer is right. AI Engineering: evaluate component by component and end to end. The parts tell you where a failure happened.

Each layer has its own tools:

Embeddings can be evaluated on their own, the book notes: good if more-similar documents get closer vectors. The MTEB benchmark tests embeddings across tasks including retrieval, classification and clustering.
Retrieval is evaluated against questions with known relevant documents: the next two pages.
Generation is evaluated on whether the answer is supported by the retrieved passages and cites them correctly.
End to end, the answer is judged like any other output, with Module 04's criteria, references and judges.

The book's last word on retrievers: a retriever is good if it helps the system generate high-quality answers. Component metrics are for diagnosis; the end-to-end result is what users see.

Quick check

Why measure retrieval separately if end-to-end accuracy is what matters?

Retrieval metrics are easier to raiseBecause end-to-end accuracy can't be measuredTo know where a failure happened, so you change the right componentRetrieval metrics replace end-to-end evaluation