Topic 5

Throughput: everyone vs each user

Latency is one request's experience. ThroughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught is how much work the system gets through: usually output tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught per second across all requests. AI Engineering (Ch. 9) recommends counting input (prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught) and output (decode) throughput separately, because the two phases are limited by different things, and notes that per-user tokens/s and completed requests per minute are also used.

Throughput is where cost comes from: a machine costs the same per hour whether it serves 100 tokens a second or 1,000.

On the GPU, the same move takes total throughput from 179 to 1,349 tokens/s while each user drops from 179 to 84. Both devices show the same trade-off: batching raises throughput and lowers per-user speed. AI Engineering (Ch. 9) cites a LinkedIn report that doubling or tripling throughput is common if you accept worse TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught and TPOTTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught.

Why does running 16 requests cost so much less than 16 times one request? That is the subject of lesson 2: during decode the processor spends most of its time reading the model's weights, and a batch reads them once for everyone.

Throughput across models

Tokens are not a common currency: different models split the same text into different numbers of tokens (Token counts are not word counts). The book's advice is to compare systems on cost per requestCost per requestWhat one request costs end to end: input tokens × input price + output tokens × output price for an API, or machine time × hourly price when self-hosting. Comparable across models with different tokenizers, unlike a price per token.Where this is taught, not tokens per second, when the tokenizersTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught differ.