Senior interview question
What is criteria drift, and what does it mean for your evaluation?
Model answer
You need criteria to grade outputs, but grading outputs is how you discover what the criteria should be. So the rubric is never finished: add criteria as new failure modes appear, version it, and re-score a baseline after each change so results stay comparable.
Practise the rest
7 Error analysis questions, 492 in all, each with a model answer. Free.
More Error analysis interview questions
- What is error analysis?
- Your pass rate dropped five points this week. What do you do first?
- How do you decide which failure mode to fix first?
- How do you make sure a fixed failure doesn't come back?
- A RAG assistant gave a wrong answer. How do you find the cause?
- A recommender ranked mostly by distance raised add-to-cart rate but lowered revenue per session: diagnose why and design a replacement with experiments to separate causes.