Senior interview question
How many test cases do you need to compare two LLM systems?
Model answer
Start from the smallest difference that would change your decision.
Scroll sideways to see the whole diagram
To be about 95% confident one system is better: a 30-point gap shows up in about 10 cases.
Text version
- To be about 95% confident one system is better: a 30-point gap shows up in about 10 cases.
- A 10-point gap needs about 100.
- A 3-point gap needs about 1,000, and a 1-point gap about 10,000.
- The pattern: every time the difference you care about shrinks 3×, the cases you need grow about 10×. A rough guide, reported from OpenAI.
A rough rule reported from OpenAI: about 100 cases to detect 10 points, about 1,000 for 3, about 10,000 for 1. Every 3× smaller difference needs about 10× more cases. If a 2-point gain is worth real money, budget for thousands of cases.
Practise the rest
7 Comparing systems questions, 492 in all, each with a model answer. Free.
More Comparing systems interview questions
- Why does every evaluation result need a baseline?
- What is an ablation study, and when would you run one?
- Version B scores 4 points higher than A on 100 test cases. Is B better?
- What is benchmark contamination, and how would you detect it?
- How should you use public leaderboards like Chatbot Arena when choosing a model?
- You only have small counts. How would you statistically compare two rare-event detection models and decide which is better?