Goodput
The rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.
Interview questions about Goodput
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each…