Junior interview question
Why report p50 and p99 latency rather than the mean?
Model answer
Latency is a distribution, and the mean hides the tail. The same request sent 200 times on our laptop CPU had a p50 of 61.3 ms, a p99 of 112.8 ms and a slowest run of 176.2 ms. Users meet the tail: send 100 messages a day and you hit the p99 about daily.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
- Explain how a server like vLLM runs a decoder-only model efficiently.