Topic 1

What users wait for

In Module 01 you split a reply into prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught, one parallel pass over the prompt, and decode, one tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught at a time (Prefill and decode). This lesson is about measuring that properly: what to time, how to summarise many requests, and which numbers a product should actually promise.

Start with the user. They don't feel "the model's speed". They feel two waits: how long until something appears, and how long until the answer is complete.

Press Replay and watch the ticks arrive at their measured times (10× slower, so you can see them). Then switch to Apple GPU: the same reply on the same laptop's GPU starts at 26 ms and finishes at 382 ms.

Two waits, two names

• Time to first tokenTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught (TTFT): from sending the request to the first token arriving. It is mostly prefill, so it grows with the prompt.
• Total latency: until the last token arrives. For a long answer it is mostly decode.

AI Engineering (Ch. 9, "Inference Performance Metrics") points out that two systems with the same total latency can feel very different: one starts instantly and trickles, the other pauses and then flies. Which one users prefer is a product question, worth testing with real users.

The same section gives a useful ceiling for the trickle: a very fast reader gets through roughly one token every 120 ms. Our CPU produced one every 8.6 ms, about 14 times faster than anyone can read. Once streaming, extra decode speed stops mattering to the reader. It still matters for total latency, for answers nobody reads live (JSON for another program, say), and for cost.