Lesson 2 · 6 topics · 20 min · intermediate

Where the Time Goes: Memory, Compute and Batching

You will be able to

Tell compute-bound from memory bandwidth-bound work, and explain operations per byte (arithmetic intensity).
Compute the bytes a decode step reads and the bandwidth it achieves, and compare it with a measured ceiling (MBU).
Read a roofline built from measured ceilings, and place decode and prefill on it.
Explain why batching raises throughput: one read of the weights serves every sequence in the batch.
Estimate how many conversations fit in accelerator memory from weights plus KV cache.
Compare static, dynamic and continuous batching on a simulated workload, stating the assumptions.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 9 "Inference Optimization": "Computational bottlenecks", "Utilization, MFU, and MBU", sidebar "Calculating the KV Cache Size", "Batching".
Roofline: An Insightful Visual Performance Model for Multicore Architectures · Samuel Williams et al., 2009
Samuel Williams, Andrew Waterman, David Patterson. Communications of the ACM 52(4). Used: Whole article. Open source
Orca: A Distributed Serving System for Transformer-Based Generative Models · Gyeong-In Yu et al., 2022
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun. OSDI '22. Used: Abstract (iteration-level scheduling, selective batching). Open source
Apple unleashes M5, the next big leap in AI performance for Apple silicon · Apple Newsroom, 2025
Apple Newsroom. Apple. Used: "M5 offers unified memory bandwidth of 153GB/s". Open source

You will be able to

Tell compute-bound from memory bandwidth-bound work, and explain operations per byte (arithmetic intensity).

Compute the bytes a decode step reads and the bandwidth it achieves, and compare it with a measured ceiling (MBU).

Read a roofline built from measured ceilings, and place decode and prefill on it.

Explain why batching raises throughput: one read of the weights serves every sequence in the batch.

Estimate how many conversations fit in accelerator memory from weights plus KV cache.

Compare static, dynamic and continuous batching on a simulated workload, stating the assumptions.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 9 "Inference Optimization": "Computational bottlenecks", "Utilization, MFU, and MBU", sidebar "Calculating the KV Cache Size", "Batching".

Roofline: An Insightful Visual Performance Model for Multicore Architectures · Samuel Williams et al., 2009

Samuel Williams, Andrew Waterman, David Patterson. Communications of the ACM 52(4). Used: Whole article. Open source

Orca: A Distributed Serving System for Transformer-Based Generative Models · Gyeong-In Yu et al., 2022

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun. OSDI '22. Used: Abstract (iteration-level scheduling, selective batching). Open source

Apple unleashes M5, the next big leap in AI performance for Apple silicon · Apple Newsroom, 2025

Apple Newsroom. Apple. Used: "M5 offers unified memory bandwidth of 153GB/s". Open source