Senior interview questionAsked at NVIDIA · interview write-up
Design optimised inference pipelines, and tell me how you would benchmark them.
Model answer
Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt. Report time to first token, inter-token latency and end-to-end at p50 and p99, plus throughput at a latency SLO; peak throughput at unbounded latency means little.
Scroll sideways to see the whole diagram
Decide what is held fixed. Two numbers are comparable only if everything except the thing you are testing is the same.
Text version
- Decide what is held fixed. Two numbers are comparable only if everything except the thing you are testing is the same.
- Throw away the first runs. Our scripts ran two or three untimed requests before timing anything.
- Time the same request many times. One run is one draw from a distribution.
- Summarise with percentiles. A few slow runs move the mean without telling you how slow they were.
- Write down everything you didn't vary, so someone else (or you, next month) can tell whether a new number is comparable.
After AI Engineering, Ch. 9, "Inference Performance Metrics", and the measuring routine in scripts/generate-latency-data.py.
Change one knob at a time, batch size, precision, tensor-parallel degree, engine, and rerun a quality eval after each, since quantization can quietly cost accuracy. Warm up, pin versions, repeat runs, and profile to explain each result.
Practise the rest
11 Benchmarking questions, 492 in all, each with a model answer. Free.
More from NVIDIA interview questions.
More Benchmarking interview questions
- What does a fair inference benchmark hold fixed?
- An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?
- A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
- In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?
- What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?