Percentile (p50, p99)
Also called percentiles, percentile, p99, p50
pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.
Interview questions about Percentile (p50, p99)
- Why report p50 and p99 latency rather than the mean?
Latency is a distribution, and the mean hides the tail. The same request sent 200 times on our laptop CPU had a p50 of 61.3 ms, a p99 of 112.8 ms and a…
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
Measure it myself: their number is their workload on their day.
- Your AI chatbot takes too long to respond. What do you do?
First measure which wait it is: time to first token or total time, at p50 and p99. Slow first token: shorten the prompt, cache the static prefix, cut queueing.
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- What does a fair inference benchmark hold fixed?
Everything except the thing you changed: model and precision, prompt and reply lengths, batch, concurrency and arrival rate, warm-up, repeats, environment and…
- Design optimised inference pipelines, and tell me how you would benchmark them.
Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt.
- Your answers must start within one second. How do you budget it?
Add up the stages before the first token: router, retrieval, reranker, the model's time to first token.
- In the recall stage of a recommender, design a model that chooses how many candidates K to retrieve per request given latency and compute cost.
Predict the marginal value of more candidates per request. Label offline by replaying traffic at several K values and recording where the final ranker's top…