Error analysis interview questions
7 questions with model answers
- junior 1
- mid 4
- senior 2
7 error analysis questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 4 mid-level and 2 senior. The lesson Error Analysis: Finding What Goes Wrong teaches the ideas behind them.Updated
Model answer
Reading real failures, grouping them into failure modes, counting each, fixing the one that matters most, and turning those failures into test cases.
Scroll sideways to see the whole diagram
Start with real outputs: failing test cases, and a sample of live traffic.
Text version
- Start with real outputs: failing test cases, and a sample of live traffic.
- Read them, one by one, and write down what went wrong in plain words.
- Only then group the notes into failure modes, and count how often each happens.
- Pick what to fix from how often each failure happens, how much harm it does, and what fixing it costs.
- Keep the failing examples as test cases, so the fix stays fixed.
- Then repeat: the next round finds the next most important failure.
Metrics tell you how often something goes wrong; reading the outputs tells you what goes wrong.
Model answer
Read a sample of the new failures and note what went wrong in each, in plain words, before trying a fix. Then group the notes and count them. Read some passing outputs too: scorers and judges can pass things for the wrong reason.
Model answer
Weigh how often it happens by how much harm it does, and both against the cost of the fix. A rare failure that gives customers wrong prices can outrank a common one that only makes them ask again. Write the severities down and agree on them, like a rubric.
Model answer
Turn it into a test case: the original input, the expected answer, and a check a program can run, in a known-failures set that runs on every change. Keep that set separate from the production-like one, or a growing pile of hard cases makes an improving system look worse.
Model answer
Check the steps in order, from the logs.
Scroll sideways to see the whole diagram
An answer was wrong. Before fixing anything, find out where.
Text version
- An answer was wrong. Before fixing anything, find out where.
- If the right passage never reached the model, no prompt change will help: fix retrieval.
- If it was there but the answer ignored or contradicted it, fix the prompt or the model.
- If the answer followed a passage that was itself wrong or out of date, fix the source.
- All three questions need the retrieved passages and the prompt logged with every answer.
Was the right passage retrieved? If not, fix retrieval. Was it there but ignored or contradicted? Fix the prompt or the model. Was the source itself wrong or stale? Fix the docs. Without each answer's retrieved passages and prompt version logged, you can't tell which.
Model answer
You need criteria to grade outputs, but grading outputs is how you discover what the criteria should be. So the rubric is never finished: add criteria as new failure modes appear, version it, and re-score a baseline after each change so results stay comparable.
Model answer
Sorting by distance favours cheap, nearby items, so carts fill more often but with smaller baskets. Decompose revenue per session into conversion times order value and slice by category and price band to see which term moved.
Replace it with a ranker scoring expected revenue: relevance times conversion times basket value, with distance as one feature. To separate causes, A/B test old, new, and a distance ranking reweighted to the old price mix.