Topic 2
Measuring a stream
Every latency number in this lesson comes from one simple record: the time each token arrives. Take a clock reading when the request is sent, then another every time a token comes back. Everything else is arithmetic on that list.
Scroll sideways to see the whole diagram
Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
Text version
- Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
- A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
- Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
- Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
- The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.
Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.
From the arrival times t1 … tN of N tokens, measured from the moment of sending (t0):
- TTFT = t1 − t0.
- Inter-token latency (ITL) = each gap tk − tk−1. AI Engineering (Ch. 9) mentions time between tokens (TBT) as another name for the same measurement.
- Time per output token (TPOT) = the average of those gaps.
- Total latency = tN − t0 ≈ TTFT + TPOT × (N − 1). The book writes × N; the difference is the first token, which is already inside TTFT.
Predict first
Measuring honestly
Three traps make timings wrong without any error message:
- Warm-up. The first call is slow: weights load into caches, and GPU code gets compiled the first time it runs. We ran three warm-up requests before timing anything.
- Asynchronous hardware. A GPU call returns before the GPU has finished. Stop the clock then, and you time the request to start work, not the work. Our script reads each token back to the CPU before taking the timestamp, which forces the wait.
- Where the clock sits. Timing inside the server leaves out the network and the queue. Timing in the client includes them, which is what users feel. Say which one you measured.
Our numbers in this lesson are all from one laptop (Apple M5), GPT-2 small, one request at a time, with nothing in between. A hosted model adds network and queueing on top.
Quick check