Senior interview question
How do you make a RAG system say “I don't know”, and how do you know it's working?
Model answer
Two guards: a similarity threshold so weak matches never reach the prompt, and instructions to answer only from the passages and say when they don't contain the answer.
Then measure both sides: abstention on questions you know are unanswerable, and refusals on answerable ones. Every guard trades hallucinated answers for false refusals.
Practise the rest
6 RAG evaluation questions, 492 in all, each with a model answer. Free.
More RAG evaluation interview questions
- What are context precision and context recall?
- Two systems each retrieve one relevant result in their top three, at positions 1 and 3. How do rank-aware metrics tell them apart?
- Why score retrieval and answers separately?
- What's the difference between faithfulness and citation precision?
- Your RAG answers score faithfulness 1.0, yet users report wrong answers. How?