Topic 4
Retrieval and answers, scored together
Good retrieval metrics don't guarantee good answers, and a good answer doesn't prove retrieval worked. Scoring both on the same questions sorts every result into one of four cells.
Predict first
Scroll sideways to see the whole diagram
Score each test question twice: was the relevant chunk retrieved, and was the answer right?
Text version
- Score each test question twice: was the relevant chunk retrieved, and was the answer right?
- Retrieved and right: the system worked the way it was designed to.
- Retrieved but wrong: the evidence was in the prompt, so the fix is on the generation side.
- Missed and wrong: retrieval failed. A better generator can't fix this.
- Missed but right: the model answered from memory. End-to-end accuracy counts it, but the answer can't be checked against a source, and it may not hold when the facts change.
Each cell points somewhere different:
- Retrieved, right: working as designed.
- Retrieved, wrong: the evidence was in the prompt. The generator, the prompt or the presentation of the passages is at fault. Lesson 1 called this the generator's ceiling.
- Missed, wrong: a retrieval failure. Lessons 2 to 4 are the toolbox.
- Missed, right: an answer without evidence. Track how often it happens; a system that's often right this way is relying on the model's memory, not your documents.
Cost and latency belong in the comparison too. AI Engineering (Ch. 6, "Comparing retrieval algorithms") notes that much of a RAG system's latency comes from generating the output, so query embedding and vector search may add little by comparison, but it still affects the user experience. And it repeats the cost point from lesson 2: embedding, storing and searching vectors costs money, especially when data changes often.
Quick check