Mid-level interview questionAsked by candidates · YouTube
Your AI chatbot takes too long to respond. What do you do?
Model answer
First measure which wait it is: time to first token or total time, at p50 and p99.
Scroll sideways to see the whole diagram
Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
Text version
- Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
- A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
- Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
- Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
- The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.
Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.
Slow first token: shorten the prompt, cache the static prefix, cut queueing. Slow total: cap output length, use a smaller or quantized model, check batching. Always stream, so users read while it writes, and fix the biggest measured part first.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?