Measuring Inference: Latency, Throughput and Goodput

You will be able to

• Explain why streaming changes what a user waits for, and separate time to first token, time per output token and total latency in a measured stream.
• Measure a streamed reply by recording one timestamp per token, and derive TTFT, inter-token latency, TPOT and total latency from those timestamps.
• Treat latency as a distribution: read p50, p90 and p99 from 200 repeated runs, and explain why the mean misleads when there is a tail.
• Estimate a request's latency from measured prompt-length curves, stating the assumptions, and compare how much faster hardware helps short and long prompts.
• Distinguish total throughput from per-user throughput, and read the batching trade-off from measured CPU and GPU numbers.
• Define goodput against a latency objective (SLO) and compute it for a measured workload; explain why hidden tokens make time to publish differ from TTFT.

Topics

1. 1What users wait for
2. 2Measuring a stream
3. 3Latency is a distribution
4. 4Prompt length and hardware
5. 5Throughput: everyone vs each user
6. 6Goodput: counting only what meets the target
7. 7Hidden tokens, and wrap-up