Senior interview question
A RAG assistant gave a wrong answer. How do you find the cause?
Model answer
Check the steps in order, from the logs.
Scroll sideways to see the whole diagram
An answer was wrong. Before fixing anything, find out where.
Text version
- An answer was wrong. Before fixing anything, find out where.
- If the right passage never reached the model, no prompt change will help: fix retrieval.
- If it was there but the answer ignored or contradicted it, fix the prompt or the model.
- If the answer followed a passage that was itself wrong or out of date, fix the source.
- All three questions need the retrieved passages and the prompt logged with every answer.
Was the right passage retrieved? If not, fix retrieval. Was it there but ignored or contradicted? Fix the prompt or the model. Was the source itself wrong or stale? Fix the docs. Without each answer's retrieved passages and prompt version logged, you can't tell which.
Practise the rest
7 Error analysis questions, 492 in all, each with a model answer. Free.
More Error analysis interview questions
- What is error analysis?
- Your pass rate dropped five points this week. What do you do first?
- How do you decide which failure mode to fix first?
- How do you make sure a fixed failure doesn't come back?
- What is criteria drift, and what does it mean for your evaluation?
- A recommender ranked mostly by distance raised add-to-cart rate but lowered revenue per session: diagnose why and design a replacement with experiments to separate causes.