Mid-level interview questionAsked at Citadel · interview write-up
Your LLM gives different answers to the same input. How would you make inference reproducible, and how many repeated runs would you need for a stable score?
Model answer
First remove the randomness you control: greedy decoding or a fixed seed, and a pinned model, prompt and parameters. Even then, GPU serving isn't bitwise deterministic, because batch composition changes floating-point reduction order, so cache outputs for exact replay.
For the score, treat runs as samples. Measure the standard deviation across repeats in a pilot, then pick n so the standard error, sd over root n, is well below the difference you need to detect.
Practise the rest
11 Benchmarking questions, 492 in all, each with a model answer. Free.
More from Citadel interview questions.
More Benchmarking interview questions
- What does a fair inference benchmark hold fixed?
- An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?
- A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
- In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?
- What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?