Senior interview question
A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?
Model answer
Ask: for which phase, prompt length and batch? On our test laptop the Apple GPU's first-token advantage was 1.6× at 16 tokens and 2.3× at 960, but batch-1 decode only about 1.4×, which fits bandwidth-bound decode with similar copy ceilings. Short prompts with long replies are almost all decode, so the headline barely applies.
The book's three questions: can it run the workload, how long does it take, what does it cost. I'd measure my own prefill and decode mix, comparing ratios from one run.
Practise the rest
11 Benchmarking questions, 492 in all, each with a model answer. Free.
More Benchmarking interview questions
- What does a fair inference benchmark hold fixed?
- An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
- In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?
- What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?
- Design optimised inference pipelines, and tell me how you would benchmark them.