Senior interview questionAsked at Anthropic · interview write-up
Design the routing and scheduling layer in front of an inference fleet that serves many tenants.
Model answer
Three jobs. Admission: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch. Fair scheduling inside each replica: weighted fair queuing across tenants, so one heavy tenant can't fill every batch slot. Routing: choose replicas by load and prefix-cache affinity, so repeated system prompts hit warm KV cache, spilling over when a replica runs hot.
Scroll sideways to see the whole diagram
Put what never changes first. The cache can reuse everything up to the first token that differs.
Text version
- Put what never changes first. The cache can reuse everything up to the first token that differs.
- Put something that changes at the top, like a date, and the match ends at token one: nothing is reused.
- This is not a provider quirk. Through attention, each position's keys and values depend on everything before it, so only an identical prefix gives identical values.
Matching rule: Anthropic and OpenAI prompt-caching docs (fetched 2026-10-21). Illustrative prompts.
Track time to first token and per-token latency per tenant against SLOs, and under overload shed the lowest priority first with retry-after errors.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Anthropic interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?