Mid-level interview question
What is benchmark contamination, and how would you detect it?
Model answer
A model scoring well because it saw the test in training, usually by accident through web scraping. Detect it with n-gram overlap against the training data, accurate but needing the data and compute, or unusually low perplexity on the test, cheaper and less accurate. For your own decisions, a private test set avoids it.
Practise the rest
7 Comparing systems questions, 492 in all, each with a model answer. Free.
More Comparing systems interview questions
- Why does every evaluation result need a baseline?
- What is an ablation study, and when would you run one?
- Version B scores 4 points higher than A on 100 test cases. Is B better?
- How many test cases do you need to compare two LLM systems?
- How should you use public leaderboards like Chatbot Arena when choosing a model?
- You only have small counts. How would you statistically compare two rare-event detection models and decide which is better?