Topic 3

Latency is a distribution

Send the same request twice and you get two different times. Send it 200 times and you get a distribution: what happens to the processor and memory at that moment shifts each result a little, and occasionally a lot.

Move the percentilePercentile (p50, p99)pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.Where this is taught slider. p50 (the median) is the time half the requests beat: 61.3 ms. p90 is 70.3 ms. p99 is 112.8 ms: one request in a hundred was slower than that. Now switch to the Apple GPU: the same request is tight, from 26.0 to 28.3 ms. A different processor, a different shape.

Why not just the average?

The mean, 65.0 ms here, sits above the median: a few slow runs drag it up, yet it tells you nothing about how slow they were. AI Engineering (Ch. 9) gives a starker example: nine requests near 100 ms and one at 3 seconds average out to almost 400 ms, a number that matches no request at all. The book recommends watching percentiles, typically p50, p90, p95 and p99, and looking into the outliers, since they are often symptoms of something wrong.

The tail matters more than its size suggests. A page that makes several model calls is as slow as its slowest call, and a user who sends 100 messages a day will hit the p99 about once a day.