Senior interview questionAsked at Anthropic · interview write-up
Design request batching for GPU inference, including scheduling, autoscaling and routing.
Model answer
Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long ones. The scheduler admits work while KV-cache memory allows, using paged allocation, and chunks long prefills so they don't stall decodes.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
Route by queue depth and free cache, preferring a replica that holds the prompt prefix. Autoscale on queue length and time to first token, not GPU utilisation, keep warm spares because loading weights is slow, and shed batch traffic first under overload.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Anthropic interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?