Senior interview question
Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
Model answer
Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each user dropped to 54.
So I'd decide on goodput: requests per minute meeting the SLO, say first token within 100 ms and at most 10 ms per token. Throughput also counts requests users gave up on, so goodput can fall while it rises. On a 120-request CPU workload, 83 met that SLO.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Explain how a server like vLLM runs a decoder-only model efficiently.