Mid-level interview question
You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
Model answer
GPU calls are asynchronous: they return before the GPU finishes, so I timed the queuing, not the work. Wait for a result, or synchronise the device, before stopping the clock.
Two more traps give no error: skipping warm-up, since first calls compile code and fill caches; and where the clock sits, since server-side timing leaves out the network and queue users feel.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
- Explain how a server like vLLM runs a decoder-only model efficiently.