Comparing systems interview questions
7 questions with model answers
- junior 2
- mid 3
- senior 2
7 comparing systems questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 2 junior, 3 mid-level and 2 senior. The lesson Comparing Systems You Can Trust teaches the ideas behind them.Updated
Model answer
A score means little until you know what it beats. Random guessing is the floor (25% on four-option multiple choice), then the simplest approach that could work, the version you run today, and sometimes a person doing the job. Most real decisions are against what you'd replace.
Model answer
Remove one part at a time, such as retrieval, a reranker or prompt examples, and re-run the same evaluation. The drop is what that part is worth. A part whose removal costs nothing measurable can go: it's pure cost in tokens, latency and upkeep. Change one thing at a time, or you can't tell which change mattered.
Model answer
Not yet: bootstrap it. Resample the 100 cases with replacement many times, the same cases for both versions, and look at B's lead each time. If the 95% interval includes zero, the set can't tell them apart. Four points on 100 cases is four cases; on 1,000 cases the same lead can be clearly real.
Model answer
A model scoring well because it saw the test in training, usually by accident through web scraping. Detect it with n-gram overlap against the training data, accurate but needing the data and compute, or unusually low perplexity on the test, cheaper and less accurate. For your own decisions, a private test set avoids it.
Model answer
Start from the smallest difference that would change your decision.
Scroll sideways to see the whole diagram
To be about 95% confident one system is better: a 30-point gap shows up in about 10 cases.
Text version
- To be about 95% confident one system is better: a 30-point gap shows up in about 10 cases.
- A 10-point gap needs about 100.
- A 3-point gap needs about 1,000, and a 1-point gap about 10,000.
- The pattern: every time the difference you care about shrinks 3×, the cases you need grow about 10×. A rough guide, reported from OpenAI.
A rough rule reported from OpenAI: about 100 cases to detect 10 points, about 1,000 for 3, about 10,000 for 1. Every 3× smaller difference needs about 10× more cases. If a 2-point gain is worth real money, budget for thousands of cases.
Model answer
As a shortlist, not a decision. Rankings come from crowd votes on many simple prompts, with few votes per pair, and they assume transitivity. Elo even depends on match order, one reason Arena moved to Bradley–Terry. And a win rate says which is better, not whether either is good enough for your tasks, in your setup, at its cost.
Model answer
Accuracy hides rare events; compare counts on the same cases. If both models saw the same data, use McNemar's paired test, which uses only cases where they disagree. For rates like detections per thousand miles, compare Poisson rates with an exact test. With few events, report the wide intervals too.