Time to first token (TTFT)
Also called time to first token, TTFT
How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.
Interview questions about Time to first token (TTFT)
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
Record one timestamp per token, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total…
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
TTFT counts the first generated token; the user waits for the first visible one, after every hidden plan or reasoning token, plus any tool calls.
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
Measure it myself: their number is their workload on their day.
- Users say your summariser 'takes ages to start' on long documents but then writes quickly. Explain what's happening in terms of prefill and decode.
The wait is prefill. Prefill runs the whole prompt in one parallel pass and sets the time to first token; decode then writes one token per step, each costing…
- Request A has a 960-token prompt and 10 output tokens. Request B has a 16-token prompt and 200 output tokens. Same model and machine: which finishes first, and what does that tell you about a chat app that resends the full history every turn?
A. On our GPT-2 small measurements, A is about 220 ms of prefill plus 10 steps of about 10 ms, roughly 0.3 s; B is about 15 ms plus 200 sequential steps of…
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
Time to first token (TTFT), mostly prefill, and total latency, mostly decode.
- Your AI chatbot takes too long to respond. What do you do?
First measure which wait it is: time to first token or total time, at p50 and p99. Slow first token: shorten the prompt, cache the static prefix, cut queueing.
- Design request batching for GPU inference, including scheduling, autoscaling and routing.
Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long…