Mid-level interview question
How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
Model answer
Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline. On our test laptop's CPU, a batch-1 decode step moved 507 MB in 9.1 ms: 55.7 GB/s, 46% of the copy ceiling, at about 0.5 operations per byte, far left of the ridge at 16. A 960-token prefill reached 51% of the matrix-multiply ceiling. For decode, more bandwidth or fewer bytes helps; more FLOP/s doesn't.
Practise the rest
10 Where the time goes questions, 492 in all, each with a model answer. Free.
More Where the time goes interview questions
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
- You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?
- Explain the optimisations an ML compiler makes, such as kernel fusion, and how they map to the hardware.