Topic 1
Two kinds of slow
Lesson 1 measured how long inference takes. This lesson asks why, and the answer decides which fix works.
AI Engineering (Ch. 9, "Computational bottlenecks") names two kinds of slow:
- Compute-bound: the time is set by how much arithmetic there is. A faster chip, or more chips, helps.
- Memory bandwidth-bound: the time is set by how fast data moves from memory to the arithmetic units. A chip with more bandwidth helps; more arithmetic does not. People often shorten this to memory-bound, though the book notes some use that word for running out of memory instead.
Which one a piece of work is depends on its arithmetic intensity: operations done per byte read from memory. The idea comes from the Roofline paper (Williams, Waterman and Patterson, 2009).
Scroll sideways to see the whole diagram
Every forward pass has to bring the model's weights from memory to the arithmetic units: about 498 MB for GPT-2 small in 32-bit floats.
Text version
- Every forward pass has to bring the model's weights from memory to the arithmetic units: about 498 MB for GPT-2 small in 32-bit floats.
- A decode step does that whole read to produce one token for one sequence, plus that sequence's KV cache. About 2 operations per weight, 4 bytes per weight: half an operation per byte.
- Decode 8 sequences together and the same read serves all 8. Only the KV caches grow with the batch.
- Prefill pushes every prompt token through in one pass, so one read of the weights serves all 960.
- Operations per byte decides the limit. Few, and the chip waits on memory; many, and it waits on arithmetic. The next pages measure both on one laptop.
GPT-2 small (124,439,808 weights, float32); KV cache from the Module 01 formula at a 128-token context.
Predict first
The book says prefill is compute-bound and decode is bandwidth-bound. Module 01 noted that its timings could not test this. The next page does.