# Comparing systems interview questions > 7 comparing systems interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/comparing-systems Last updated: 2026-10-23 7 questions: 2 junior, 3 mid-level and 2 senior. Taught in [Comparing Systems You Can Trust](https://llmdojo.dev/learn/evaluation-experimentation/comparing-systems). ## [Why does every evaluation result need a baseline?](https://llmdojo.dev/interview/q/why-does-every-evaluation-result-need-a-baseline) Level: Junior A score means little until you know what it beats. **Random guessing** is the floor (25% on four-option multiple choice), then the **simplest approach** that could work, the **version you run today**, and sometimes a person doing the job. Most real decisions are against what you'd replace. ## [What is an ablation study, and when would you run one?](https://llmdojo.dev/interview/q/what-is-an-ablation-study-and-when-would-you-run-one) Level: Mid-level Remove **one part at a time**, such as retrieval, a reranker or prompt examples, and re-run the same evaluation. The drop is what that part is worth. A part whose removal costs nothing measurable can go: it's pure cost in tokens, latency and upkeep. Change one thing at a time, or you can't tell which change mattered. ## [Version B scores 4 points higher than A on 100 test cases. Is B better?](https://llmdojo.dev/interview/q/version-b-scores-4-points-higher-than-a-on-100-test-cases) Level: Mid-level Not yet: **bootstrap** it. Resample the 100 cases with replacement many times, the same cases for both versions, and look at B's lead each time. If the **95% interval includes zero**, the set can't tell them apart. Four points on 100 cases is four cases; on 1,000 cases the same lead can be clearly real. ## [What is benchmark contamination, and how would you detect it?](https://llmdojo.dev/interview/q/what-is-benchmark-contamination-and-how-would-you-detect-it) Level: Mid-level A model scoring well because it **saw the test in training**, usually by accident through web scraping. Detect it with **n-gram overlap** against the training data, accurate but needing the data and compute, or unusually **low perplexity** on the test, cheaper and less accurate. For your own decisions, a **private test set** avoids it. ## [How many test cases do you need to compare two LLM systems?](https://llmdojo.dev/interview/q/how-many-test-cases-do-you-need-to-compare-two-llm-systems) Level: Senior Start from the **smallest difference that would change your decision**. **Diagram: How many cases to see a difference?** 1. To be about 95% confident one system is better: a 30-point gap shows up in about 10 cases. 2. A 10-point gap needs about 100. 3. A 3-point gap needs about 1,000, and a 1-point gap about 10,000. 4. The pattern: every time the difference you care about shrinks 3×, the cases you need grow about 10×. A rough guide, reported from OpenAI. A rough rule reported from OpenAI: about 100 cases to detect 10 points, about 1,000 for 3, about 10,000 for 1. Every **3× smaller difference needs about 10× more cases**. If a 2-point gain is worth real money, budget for thousands of cases. ## [How should you use public leaderboards like Chatbot Arena when choosing a model?](https://llmdojo.dev/interview/q/how-should-you-use-public-leaderboards-like-chatbot-arena-when-choosing-a) Level: Senior As a **shortlist**, not a decision. Rankings come from crowd votes on many simple prompts, with few votes per pair, and they assume transitivity. Elo even depends on **match order**, one reason Arena moved to Bradley–Terry. And a win rate says which is better, not whether either is **good enough** for your tasks, in your setup, at its cost. ## [You only have small counts. How would you statistically compare two rare-event detection models and decide which is better?](https://llmdojo.dev/interview/q/you-only-have-small-counts-how-would-you-statistically-compare-two-rare) Level: Junior Accuracy hides rare events; compare **counts on the same cases**. If both models saw the same data, use **McNemar's** paired test, which uses only cases where they disagree. For rates like detections per thousand miles, compare Poisson rates with an exact test. With few events, report the wide intervals too. ## Related - [All interview questions](https://llmdojo.dev/interview)