Senior interview question
Your RAG answers score faithfulness 1.0, yet users report wrong answers. How?
Model answer
Faithfulness measures loyalty to the evidence, not the evidence. A perfect summary of an outdated or irrelevant passage scores 1.0.
Scroll sideways to see the whole diagram
First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
Text version
- First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
- Is it in a chunk that says what it's about? If it was split or lost its context, fix chunking (lesson 4).
- Did retrieval return the chunk at all? If not, look at the retrievers and the query (lessons 2–4).
- Was it returned but cut: ranked below k, or over the token budget? Look at reranking and the pool size (lesson 3).
- Only if it was in the prompt is it a generation problem. Logging every step's input and output is what makes these questions answerable.
Check what was retrieved: is the source current, was the right chunk returned, is the answer relevant to the question? Pair faithfulness with retrieval metrics, answer relevance and end-to-end correctness, and audit the judge.
Practise the rest
6 RAG evaluation questions, 492 in all, each with a model answer. Free.
More RAG evaluation interview questions
- What are context precision and context recall?
- Two systems each retrieve one relevant result in their top three, at positions 1 and 3. How do rank-aware metrics tell them apart?
- Why score retrieval and answers separately?
- What's the difference between faithfulness and citation precision?
- How do you make a RAG system say “I don't know”, and how do you know it's working?