Mid-level interview questionAsked at Waymo · interview write-up
Design inference serving for a model with 100 million daily users. Give back-of-the-envelope QPS and memory for weights and caches, and explain how you'd control tail latency.
Model answer
Assume 100 million users each send 10 requests a day: about 11,600 QPS average, peaks perhaps 2 to 3 times higher. Weights are parameters times bytes, so a 70B model in 16-bit is 140 GB, sharded across GPUs. KV cache per token is 2 times layers times KV heads times head dimension times bytes, times context and concurrency.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
For tail latency: continuous batching, admission control, split prefill and decode, autoscale on queue depth.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Waymo interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?