Mid-level interview questionAsked at Anthropic · interview write-up
Design a batched LLM inference service, assuming every batch takes 100 ms to run.
Model answer
If a batch costs 100 ms whatever its size, throughput is batch size divided by 0.1 s, so bigger batches are free until memory runs out. Latency is the wait for the next batch plus 100 ms. A batcher fires when the batch fills or a timeout expires. Bound the queue and reject when full, so latency stays predictable. For long generations, batch per decode step instead.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Anthropic interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?