Evaluating RAG interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 evaluating rag questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Evaluating RAG: Retrieval, Grounding and Failures teaches the ideas behind them.Updated
Model answer
Precision: of the documents retrieved, the share that's relevant. Recall: of the relevant documents, the share retrieved.
Recall is harder in production: it needs every document judged for each query, so some frameworks report precision only.
Model answer
Average precision averages precision at each relevant position: 1/1 = 1.0 for the first system, 1/3 ≈ 0.33 for the second. MAP averages AP over queries; with one relevant document per query it equals MRR.
nDCG goes further, allowing graded relevance and discounting gains lower down.
Model answer
To know where a failure happened. Retrieved but wrong points at generation; missed and wrong at retrieval; missed but right means the model answered from memory, unverifiable.
Scroll sideways to see the whole diagram
Score each test question twice: was the relevant chunk retrieved, and was the answer right?
Text version
- Score each test question twice: was the relevant chunk retrieved, and was the answer right?
- Retrieved and right: the system worked the way it was designed to.
- Retrieved but wrong: the evidence was in the prompt, so the fix is on the generation side.
- Missed and wrong: retrieval failed. A better generator can't fix this.
- Missed but right: the model answered from memory. End-to-end accuracy counts it, but the answer can't be checked against a source, and it may not hold when the facts change.
The book: evaluate component by component and end to end.
Model answer
Faithfulness (Ragas): claims supported by any retrieved passage ÷ all claims. Citation precision: citations that support the sentence they're attached to.
A true sentence citing the wrong passage is faithful but fails citation precision. Liu et al. found only 74.5% of generative search engines' citations supported their sentence.
Model answer
Faithfulness measures loyalty to the evidence, not the evidence. A perfect summary of an outdated or irrelevant passage scores 1.0.
Scroll sideways to see the whole diagram
First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
Text version
- First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
- Is it in a chunk that says what it's about? If it was split or lost its context, fix chunking (lesson 4).
- Did retrieval return the chunk at all? If not, look at the retrievers and the query (lessons 2–4).
- Was it returned but cut: ranked below k, or over the token budget? Look at reranking and the pool size (lesson 3).
- Only if it was in the prompt is it a generation problem. Logging every step's input and output is what makes these questions answerable.
Check what was retrieved: is the source current, was the right chunk returned, is the answer relevant to the question? Pair faithfulness with retrieval metrics, answer relevance and end-to-end correctness, and audit the judge.
Model answer
Two guards: a similarity threshold so weak matches never reach the prompt, and instructions to answer only from the passages and say when they don't contain the answer.
Then measure both sides: abstention on questions you know are unanswerable, and refusals on answerable ones. Every guard trades hallucinated answers for false refusals.