Topic 4

Comparing providers and public benchmarks

With a model API you can't fix the hardware, the batch or the precision: the provider chooses them, and other customers' traffic may share the machine. So a published latency figure describes their workload on their day, not yours.

AI Engineering gives three reasons to measure for yourself:

• Latency depends on your requests. It depends on the prompt and the sampling settings, not only on the model (Ch. 4, "Evaluation Criteria" > "Cost and Latency"). The book's example criteria table measures TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught at p90 on an internal user prompt dataset, not on a public benchmark.
• Providers can change the model's behaviour. The same Llama models scored differently on benchmarks when served by different inference providers, in an experiment by Cerebras (2024). The book's explanation: a provider might use optimizations that alter a model's behaviour (Ch. 9, "Inference Optimization", Figure 9-8). Check quality alongside speed.
• Providers limit you. Some throttle a service that sends too many concurrent requests (Ch. 9, "ThroughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught and goodputGoodputThe rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.Where this is taught"), so a test at low concurrency can mislead about peak load. TokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught counts also differ by tokenizerTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught, so compare cost per requestCost per requestWhat one request costs end to end: input tokens × input price + output tokens × output price for an API, or machine time × hourly price when self-hosting. Comparable across models with different tokenizers, unlike a price per token.Where this is taught.

The book doesn't discuss public latency leaderboards. The best-known public benchmark, MLPerfMLPerf InferenceMLCommons' public inference benchmark. Its rules fix the model, dataset and accuracy target; the Server scenario reports the highest Poisson arrival rate that meets the latency limits.Where this is taught Inference, shows how strict a fair comparison is. Its rules fix the model, the dataset and accuracy targets, and define scenarios. In the Server scenario, requests arrive "according to a Poisson distribution" for at least 600 seconds, and the result is the "Maximum Poisson throughput parameter supported" within per-benchmark limits: for Llama2-70b's interactive category, TTFT 450 ms and TPOTTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught 40 ms (MLPerf Inference Rules, "Scenarios" and "Benchmarks"; rules change between rounds).