Topic 5

Capstone: the workload and its targets

The module's capstone: optimize a defined workload against latency, throughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught and cost constraints, documenting assumptions and trade-offs. Here is the workload and what it must meet. The next page is the planner.

The workload. A chat service on the laptop from lessons 1 and 2, serving GPT-2 small in float32. 600 requests arrive at random, 6 per second on average (Poisson arrivals, as in MLPerfMLPerf InferenceMLCommons' public inference benchmark. Its rules fix the model, dataset and accuracy target; the Server scenario reports the highest Poisson arrival rate that meets the latency limits.Where this is taught's Server scenario), over about 102 seconds. Prompts are 16 to 512 tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught and replies 8 to 256, drawn at random: 48,136 output tokens in all, about 473 per second.

The targets.

1. Latency: p90 time to first tokenTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught at 6 requests per second ≤ 300 ms.
2. Throughput, as capacity: once a server keeps up, its output rate is set by demand, so the real question is how much more traffic it could take. The highest arrival rate at which p90 TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught still meets 300 ms must be at least 9 per second, 1.5 times today's.
3. Memory: weights plus KV cacheKV cacheStored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.Where this is taught ≤ 2 GB. This is a budget set for the exercise; the laptop has more.
4. Cost: ≤ $0.001 per 1,000 output tokens at today's traffic, at the machine's price per hour. The price is illustrative: $1 per hour by default, editable.