Topic 6

Goodput: counting only what meets the target

ThroughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught counts every finished request, including ones so slow that the user gave up. A better question is: how many requests per minute met the latency target? That count is goodputGoodputThe rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.Where this is taught.

The target is a service level objectiveService level objective (SLO)A target a service promises to meet, such as 'first token within 100 ms and at most 10 ms per token after that'.Where this is taught (SLO), for example "first tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught within 100 ms and at most 10 ms per token after that". AI Engineering (Ch. 9, "Throughput and goodput") borrows the term from networking: goodput is the rate of requests that satisfy the SLO. The DistServe paper (Zhong et al., 2024) evaluates serving systems by the same yardstick: the highest request rate they can serve while meeting both a TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught and a TPOTTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught limit.

Each dot is one measured request: how long its first token took (across) and its average time per token (up). The shaded box is the SLO. Try:

• Tighten TTFT to 50 ms: CPU goodput falls to 61 of 120 requests, even though throughput hasn't changed.
• Switch to the Apple GPU at the original SLO: 119 of 120.
• Find the SLO the CPU passes in full, then ask whether your users would accept it.

Why it is the better target

Goodput turns a vague "faster" into a number you can optimise against, and it exposes trade-offs that throughput hides. Batching more requests raises throughput, but if it pushes requests past the SLO, goodput can fall. When we model batching in lesson 2, goodput is the number to watch.