Mid-level interview questionAsked at Apple · interview write-up
How do you measure agreement between annotators, and when would you use an LLM judge instead of human judges?
Model answer
Agreement tells you whether labels are reliable enough to score against. Raw agreement overstates it, since raters agree by chance; Cohen's kappa corrects for that with two raters, and Krippendorff's alpha handles many raters and missing labels. Low agreement often means a vague rubric.
LLM judges are cheap but biased by position, length and self-preference. Let humans build the gold set; trust a judge once its agreement with them matches human-to-human agreement.
Practise the rest
11 Benchmarking questions, 492 in all, each with a model answer. Free.
More from Apple interview questions.
More Benchmarking interview questions
- What does a fair inference benchmark hold fixed?
- An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?
- A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
- In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?
- What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?