# Evaluating RAG interview questions

> 6 evaluating rag interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/rag-evaluation
Last updated: 2026-10-23

6 questions: 1 junior, 3 mid-level and 2 senior. Taught in [Evaluating RAG: Retrieval, Grounding and Failures](https://llmdojo.dev/learn/retrieval-knowledge-systems/evaluating-rag).

## [What are context precision and context recall?](https://llmdojo.dev/interview/q/what-are-context-precision-and-context-recall)

Level: Junior

**Precision**: of the documents retrieved, the share that's relevant. **Recall**: of the relevant documents, the share retrieved.

Recall is harder in production: it needs every document judged for each query, so some frameworks report precision only.

## [Two systems each retrieve one relevant result in their top three, at positions 1 and 3. How do rank-aware metrics tell them apart?](https://llmdojo.dev/interview/q/two-systems-each-retrieve-one-relevant-result-in-their-top-three-at)

Level: Mid-level

**Average precision** averages precision at each relevant position: 1/1 = **1.0** for the first system, 1/3 ≈ **0.33** for the second. **MAP** averages AP over queries; with one relevant document per query it equals **MRR**.

**nDCG** goes further, allowing graded relevance and discounting gains lower down.

## [Why score retrieval and answers separately?](https://llmdojo.dev/interview/q/why-score-retrieval-and-answers-separately)

Level: Mid-level

To know where a failure happened. Retrieved but wrong points at generation; missed and wrong at retrieval; missed but right means the model answered from memory, unverifiable.

**Diagram: Retrieval and answer, together**

1. Score each test question twice: was the relevant chunk retrieved, and was the answer right?
2. Retrieved and right: the system worked the way it was designed to.
3. Retrieved but wrong: the evidence was in the prompt, so the fix is on the generation side.
4. Missed and wrong: retrieval failed. A better generator can't fix this.
5. Missed but right: the model answered from memory. End-to-end accuracy counts it, but the answer can't be checked against a source, and it may not hold when the facts change.

The book: evaluate component by component **and** end to end.

## [What's the difference between faithfulness and citation precision?](https://llmdojo.dev/interview/q/what-s-the-difference-between-faithfulness-and-citation-precision)

Level: Mid-level

**Faithfulness** (Ragas): claims supported by **any** retrieved passage ÷ all claims. **Citation precision**: citations that support the sentence they're attached to.

A true sentence citing the wrong passage is faithful but fails citation precision. Liu et al. found only 74.5% of generative search engines' citations supported their sentence.

## [Your RAG answers score faithfulness 1.0, yet users report wrong answers. How?](https://llmdojo.dev/interview/q/your-rag-answers-score-faithfulness-1-0-yet-users-report-wrong-answers)

Level: Senior

Faithfulness measures loyalty to the evidence, not the evidence. A perfect summary of an outdated or irrelevant passage scores 1.0.

**Diagram: Where did this RAG answer go wrong?**

1. First: is the answer in your documents, and up to date? If not, no retrieval setting will help. Fix the source or the ingestion.
2. Is it in a chunk that says what it's about? If it was split or lost its context, fix chunking (lesson 4).
3. Did retrieval return the chunk at all? If not, look at the retrievers and the query (lessons 2–4).
4. Was it returned but cut: ranked below k, or over the token budget? Look at reranking and the pool size (lesson 3).
5. Only if it was in the prompt is it a generation problem. Logging every step's input and output is what makes these questions answerable.

Check what was retrieved: is the source current, was the right chunk returned, is the answer relevant to the question? Pair faithfulness with retrieval metrics, answer relevance and end-to-end correctness, and audit the judge.

## [How do you make a RAG system say “I don't know”, and how do you know it's working?](https://llmdojo.dev/interview/q/how-do-you-make-a-rag-system-say-i-don-t-know)

Level: Senior

Two guards: a **similarity threshold** so weak matches never reach the prompt, and instructions to answer only from the passages and say when they don't contain the answer.

Then measure both sides: abstention on questions you **know** are unanswerable, and refusals on answerable ones. Every guard trades hallucinated answers for false refusals.

## Related
- [All interview questions](https://llmdojo.dev/interview)