Junior interview question
What is error analysis?
Model answer
Reading real failures, grouping them into failure modes, counting each, fixing the one that matters most, and turning those failures into test cases.
Scroll sideways to see the whole diagram
Start with real outputs: failing test cases, and a sample of live traffic.
Text version
- Start with real outputs: failing test cases, and a sample of live traffic.
- Read them, one by one, and write down what went wrong in plain words.
- Only then group the notes into failure modes, and count how often each happens.
- Pick what to fix from how often each failure happens, how much harm it does, and what fixing it costs.
- Keep the failing examples as test cases, so the fix stays fixed.
- Then repeat: the next round finds the next most important failure.
Metrics tell you how often something goes wrong; reading the outputs tells you what goes wrong.
Practise the rest
7 Error analysis questions, 492 in all, each with a model answer. Free.
More Error analysis interview questions
- Your pass rate dropped five points this week. What do you do first?
- How do you decide which failure mode to fix first?
- How do you make sure a fixed failure doesn't come back?
- A RAG assistant gave a wrong answer. How do you find the cause?
- What is criteria drift, and what does it mean for your evaluation?
- A recommender ranked mostly by distance raised add-to-cart rate but lowered revenue per session: diagnose why and design a replacement with experiments to separate causes.