Senior interview question
Version B has a higher pass rate than version A on last week's production logs. Do you ship B?
Model answer
Not on that evidence. Each version served different traffic: if B got the easy self-serve tickets, its overall rate is flattered by its mix. That's Simpson's paradox: A can win every slice and still lose overall. Compare slice by slice, or better, run both on one fixed test set with the same mix, then check the slices you care about.
Practise the rest
10 Success criteria questions, 492 in all, each with a model answer. Free.
More Success criteria interview questions
- What is evaluation-driven development?
- What kinds of criteria would you use to evaluate an LLM application?
- Two annotators grading the same chatbot replies keep disagreeing. What do you change?
- Your support bot's factual consistency is 90%. Is that good enough to automate support?
- How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?
- The prompt mixes a regression target with precision and recall: untangle it, and pick metrics and a modeling plan tied to the real decision.