Topic 4

Prompt length and hardware

TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught is mostly prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught, and prefill reads the whole prompt, so a longer prompt means a later first tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught. How much later, and how much better hardware helps, is something to measure, not guess.

The calculator interpolates between the prompt lengths we measured and adds the measured gap per output token. Its assumptions are listed under it; open them. Try:

• 256 in, 200 out: about 1.71 s on the CPU, 1.22 s on the GPU. Output dominates: decode is 96% of the CPU total.
• 960 in, 32 out: the first token alone is 223 ms on the CPU, and output is now less than three-fifths of the total.

Notice what does not change much with prompt length: the time per output tokenTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught, 7.7 to 9.7 ms on the CPU across all lengths. Long prompts mainly cost you up front.

Comparing hardware fairly

"The GPU is 2× faster" is only true for a given workload. Here the answer was 1.6× or 2.3× depending on prompt length, and the throughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught page shows it also depends on how many requests run at once. AI Engineering (Ch. 9) suggests plotting TTFT against input length rather than quoting one number; that is exactly the curve in the calculator.