Lesson 6 · 7 topics · 40 min · intermediate

Benchmarking Inference, and the Module Capstone

You will be able to

List what a fair inference benchmark holds fixed (model, precision, prompt and reply lengths, batch and concurrency, warm-up, repeats, percentiles, environment, unit) and spot a comparison that breaks one of them.
Read run-to-run variation from three measurements of the same prefill on one machine, and explain why a cause can't be named when the methods also differ.
Explain why a hardware speed-up depends on the phase, the prompt length and the batch, using measured CPU and Apple GPU ratios from the same runs.
Explain why published provider latency is not your latency, and what to measure instead: your prompts, your concurrency, p90, and quality.
Optimize a defined workload in a simulator built from measured step times against latency, capacity, memory and cost targets.
Write up the decision with its evidence, assumptions, trade-offs and the measurement that could overturn it.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 9 "Inference Optimization": "Inference Performance Metrics" > "Latency, TTFT, and TPOT" and "Throughput and goodput"; "AI Accelerators" sidebar "Selecting Accelerators"; "Inference Optimization" (Figure 9-8). Ch. 4 "Evaluate AI Systems": "Evaluation Criteria" > "Cost and Latency" (Table 4-3).
MLPerf Inference Rules · MLCommons
MLCommons. Used: Sections "Scenarios" (Server row) and "Benchmarks" (Llama2-70b row); fetched 2026-10-21. Open source

You will be able to

List what a fair inference benchmark holds fixed (model, precision, prompt and reply lengths, batch and concurrency, warm-up, repeats, percentiles, environment, unit) and spot a comparison that breaks one of them.

Read run-to-run variation from three measurements of the same prefill on one machine, and explain why a cause can't be named when the methods also differ.

Explain why a hardware speed-up depends on the phase, the prompt length and the batch, using measured CPU and Apple GPU ratios from the same runs.

Explain why published provider latency is not your latency, and what to measure instead: your prompts, your concurrency, p90, and quality.

Optimize a defined workload in a simulator built from measured step times against latency, capacity, memory and cost targets.

Write up the decision with its evidence, assumptions, trade-offs and the measurement that could overturn it.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 9 "Inference Optimization": "Inference Performance Metrics" > "Latency, TTFT, and TPOT" and "Throughput and goodput"; "AI Accelerators" sidebar "Selecting Accelerators"; "Inference Optimization" (Figure 9-8). Ch. 4 "Evaluate AI Systems": "Evaluation Criteria" > "Cost and Latency" (Table 4-3).

MLPerf Inference Rules · MLCommons

MLCommons. Used: Sections "Scenarios" (Server row) and "Benchmarks" (Llama2-70b row); fetched 2026-10-21. Open source