Topic 1

What a fair benchmark holds fixed

A benchmark answers one question: did the thing I changed make a difference? It can only answer if everything else stayed the same. Most misleading inference numbers aren't wrong; they were measured under different conditions from the number they are compared with.

The first two lessons already showed what moves the result:

• Model and precision. Bytes per weight set how much decode reads (lesson 2).
• Prompt and reply lengths. On the laptop CPU, the first tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught took 12.8 ms after a 16-token prompt and 223.3 ms after a 960-token one (lesson 1).
• Batch size, concurrency and arrival rate. Total throughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught rises with the batch while each user's tokens slow down.
• Warm-upWarm-upUntimed runs before a benchmark starts timing, so one-off costs such as loading and kernel compilation don't land in the results.Where this is taught. First calls pay one-off costs. Every script in this module runs untimed requests before timing anything.
• Repeats and percentilesPercentile (p50, p99)pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.Where this is taught. 200 identical requests gave a p50 of 61.3 ms and a p99 of 112.8 ms. Report percentilesPercentile (p50, p99)pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.Where this is taught, TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught and TPOTTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught separately, never one run or the mean.
• The environment. Hardware, threads, library versions, power state and method. The next page shows why.
• The unit. Tokens differ between tokenizersTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught, so AI Engineering (Ch. 9, "Throughput and goodputGoodputThe rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.Where this is taught") suggests comparing cost per requestCost per requestWhat one request costs end to end: input tokens × input price + output tokens × output price for an API, or machine time × hourly price when self-hosting. Comparable across models with different tokenizers, unlike a price per token.Where this is taught across systems.