Topic 7

Hidden tokens, and wrap-up

Everything so far assumed every generated tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught is shown to the user. Increasingly it isn't. A model may first write a plan, call tools, or reason step by step, and only then write the answer the user sees.

From the model's side, the first token comes as quickly as ever. From the user's side, the first visible token comes only after all the hidden ones. AI Engineering (Ch. 9) notes that some teams call this time to publishTime to publishTime until the first token the user can see. Later than time to first token when the model first generates hidden steps such as a plan.Where this is taught, to make clear it is the first token the user actually sees.

Drag the slider to change how many tokens stay hidden. The first visible token moves right one decode stepDecodeThe sequential phase of generation: one forward pass per new token.Where this is taught at a time. For an agent that calls tools between steps, the tool time is added on top.

Summary

• Users feel two waits: time to first tokenTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught, mostly prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught, and total latency, mostly decode. Streaming makes the first one what matters, and past reading speed (about 120 ms per token, from the book) tokens needn't be much faster than a reader can read.
• Record one timestamp per token; TTFT, inter-token latencyInter-token latencyThe gap between two consecutive output tokens. Its average is the time per output token.Where this is taught, TPOTTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught and total latency are all arithmetic on that list. Warm up first, wait for asynchronous hardware, and say where the clock is.
• Latency is a distribution. Report p50Percentile (p50, p99)pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.Where this is taught and p99, not the mean: the same request on our CPU ranged from 60 to 176 ms.
• TTFT grows with the prompt; time per token barely does. Hardware gains depend on the workload: our GPU was 1.6× faster at 16 tokens and 2.3× at 960.
• ThroughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught is work done per second across everyone. Batching 16 requests raised it about 7× on the CPU while each user got slower.
• GoodputGoodputThe rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.Where this is taught counts only requests that meet the SLOService level objective (SLO)A target a service promises to meet, such as 'first token within 100 ms and at most 10 ms per token after that'.Where this is taught, and is the honest target. Hidden tokens make time to publish later than TTFT.

Next

Lesson 2: where the time goes. Why decoding 16 requests at once costs far less than 16 times one, what memory bandwidth has to do with it, and a batching simulator to find the batch size that maximises goodput.