Junior interview question
How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
Model answer
Record one timestamp per token, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total latency is tN − t0, about TTFT + TPOT × (N − 1).
Scroll sideways to see the whole diagram
Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
Text version
- Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
- A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
- Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
- Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
- The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.
Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.
Keep every timestamp: our CPU stream's median gap was 8.6 ms, the slowest 11.5 ms.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
- Explain how a server like vLLM runs a decoder-only model efficiently.