Mid-level interview question
What is an ablation study, and when would you run one?
Model answer
Remove one part at a time, such as retrieval, a reranker or prompt examples, and re-run the same evaluation. The drop is what that part is worth. A part whose removal costs nothing measurable can go: it's pure cost in tokens, latency and upkeep. Change one thing at a time, or you can't tell which change mattered.
Practise the rest
7 Comparing systems questions, 492 in all, each with a model answer. Free.
More Comparing systems interview questions
- Why does every evaluation result need a baseline?
- Version B scores 4 points higher than A on 100 test cases. Is B better?
- What is benchmark contamination, and how would you detect it?
- How many test cases do you need to compare two LLM systems?
- How should you use public leaderboards like Chatbot Arena when choosing a model?
- You only have small counts. How would you statistically compare two rare-event detection models and decide which is better?