Mid-level interview question
Version B scores 4 points higher than A on 100 test cases. Is B better?
Model answer
Not yet: bootstrap it. Resample the 100 cases with replacement many times, the same cases for both versions, and look at B's lead each time. If the 95% interval includes zero, the set can't tell them apart. Four points on 100 cases is four cases; on 1,000 cases the same lead can be clearly real.
Practise the rest
7 Comparing systems questions, 492 in all, each with a model answer. Free.
More Comparing systems interview questions
- Why does every evaluation result need a baseline?
- What is an ablation study, and when would you run one?
- What is benchmark contamination, and how would you detect it?
- How many test cases do you need to compare two LLM systems?
- How should you use public leaderboards like Chatbot Arena when choosing a model?
- You only have small counts. How would you statistically compare two rare-event detection models and decide which is better?