Senior interview question
How should you use public leaderboards like Chatbot Arena when choosing a model?
Model answer
As a shortlist, not a decision. Rankings come from crowd votes on many simple prompts, with few votes per pair, and they assume transitivity. Elo even depends on match order, one reason Arena moved to Bradley–Terry. And a win rate says which is better, not whether either is good enough for your tasks, in your setup, at its cost.
Practise the rest
7 Comparing systems questions, 492 in all, each with a model answer. Free.
More Comparing systems interview questions
- Why does every evaluation result need a baseline?
- What is an ablation study, and when would you run one?
- Version B scores 4 points higher than A on 100 test cases. Is B better?
- What is benchmark contamination, and how would you detect it?
- How many test cases do you need to compare two LLM systems?
- You only have small counts. How would you statistically compare two rare-event detection models and decide which is better?