# Error analysis interview questions > 7 error analysis interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/error-analysis Last updated: 2026-10-23 7 questions: 1 junior, 4 mid-level and 2 senior. Taught in [Error Analysis: Finding What Goes Wrong](https://llmdojo.dev/learn/evaluation-experimentation/error-analysis). ## [What is error analysis?](https://llmdojo.dev/interview/q/what-is-error-analysis) Level: Junior Reading **real failures**, grouping them into **failure modes**, counting each, fixing the one that matters most, and turning those failures into test cases. **Diagram: Error analysis, one round** 1. Start with real outputs: failing test cases, and a sample of live traffic. 2. Read them, one by one, and write down what went wrong in plain words. 3. Only then group the notes into failure modes, and count how often each happens. 4. Pick what to fix from how often each failure happens, how much harm it does, and what fixing it costs. 5. Keep the failing examples as test cases, so the fix stays fixed. 6. Then repeat: the next round finds the next most important failure. Metrics tell you how often something goes wrong; reading the outputs tells you **what** goes wrong. ## [Your pass rate dropped five points this week. What do you do first?](https://llmdojo.dev/interview/q/your-pass-rate-dropped-five-points-this-week-what-do-you-do) Level: Mid-level Read a sample of the **new failures** and note what went wrong in each, in plain words, before trying a fix. Then group the notes and count them. Read some **passing** outputs too: scorers and judges can pass things for the wrong reason. ## [How do you decide which failure mode to fix first?](https://llmdojo.dev/interview/q/how-do-you-decide-which-failure-mode-to-fix-first) Level: Mid-level Weigh **how often** it happens by **how much harm** it does, and both against **the cost of the fix**. A rare failure that gives customers wrong prices can outrank a common one that only makes them ask again. Write the severities down and agree on them, like a rubric. ## [How do you make sure a fixed failure doesn't come back?](https://llmdojo.dev/interview/q/how-do-you-make-sure-a-fixed-failure-doesn-t-come-back) Level: Mid-level Turn it into a **test case**: the original input, the expected answer, and a check a program can run, in a **known-failures set** that runs on every change. Keep that set separate from the production-like one, or a growing pile of hard cases makes an improving system look worse. ## [A RAG assistant gave a wrong answer. How do you find the cause?](https://llmdojo.dev/interview/q/a-rag-assistant-gave-a-wrong-answer-how-do-you-find-the) Level: Senior Check the steps in order, from the logs. **Diagram: Which step failed?** 1. An answer was wrong. Before fixing anything, find out where. 2. If the right passage never reached the model, no prompt change will help: fix retrieval. 3. If it was there but the answer ignored or contradicted it, fix the prompt or the model. 4. If the answer followed a passage that was itself wrong or out of date, fix the source. 5. All three questions need the retrieved passages and the prompt logged with every answer. Was the **right passage retrieved**? If not, fix retrieval. Was it there but **ignored or contradicted**? Fix the prompt or the model. Was the **source itself wrong or stale**? Fix the docs. Without each answer's retrieved passages and prompt version logged, you can't tell which. ## [What is criteria drift, and what does it mean for your evaluation?](https://llmdojo.dev/interview/q/what-is-criteria-drift-and-what-does-it-mean-for-your-evaluation) Level: Senior You need criteria to grade outputs, but **grading outputs** is how you discover what the criteria should be. So the rubric is never finished: add criteria as new failure modes appear, **version** it, and re-score a baseline after each change so results stay comparable. ## [A recommender ranked mostly by distance raised add-to-cart rate but lowered revenue per session: diagnose why and design a replacement with experiments to separate causes.](https://llmdojo.dev/interview/q/a-recommender-ranked-mostly-by-distance-raised-add-to-cart-rate-but) Level: Mid-level Sorting by distance favours **cheap, nearby items**, so carts fill more often but with smaller baskets. Decompose revenue per session into conversion times **order value** and slice by category and price band to see which term moved. Replace it with a ranker scoring **expected revenue**: relevance times conversion times basket value, with distance as one feature. To separate causes, A/B test old, new, and a distance ranking reweighted to the old price mix. ## Related - [All interview questions](https://llmdojo.dev/interview)