Mid-level interview question
Why score retrieval and answers separately?
Model answer
To know where a failure happened. Retrieved but wrong points at generation; missed and wrong at retrieval; missed but right means the model answered from memory, unverifiable.
Scroll sideways to see the whole diagram
Score each test question twice: was the relevant chunk retrieved, and was the answer right?
Text version
- Score each test question twice: was the relevant chunk retrieved, and was the answer right?
- Retrieved and right: the system worked the way it was designed to.
- Retrieved but wrong: the evidence was in the prompt, so the fix is on the generation side.
- Missed and wrong: retrieval failed. A better generator can't fix this.
- Missed but right: the model answered from memory. End-to-end accuracy counts it, but the answer can't be checked against a source, and it may not hold when the facts change.
The book: evaluate component by component and end to end.
Practise the rest
6 RAG evaluation questions, 492 in all, each with a model answer. Free.
More RAG evaluation interview questions
- What are context precision and context recall?
- Two systems each retrieve one relevant result in their top three, at positions 1 and 3. How do rank-aware metrics tell them apart?
- What's the difference between faithfulness and citation precision?
- Your RAG answers score faithfulness 1.0, yet users report wrong answers. How?
- How do you make a RAG system say “I don't know”, and how do you know it's working?