Mid-level interview question
Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
Model answer
Each step reads the weights once and uses them for every sequence; only each sequence's KV cache is extra. On our test laptop's Apple GPU, a step took 6.3 ms for one sequence and 9.3 ms for 8, so throughput rose from 159 to 860 tokens/s.
Limits: the longer step slows each user, and every conversation's KV cache needs memory.
Practise the rest
10 Where the time goes questions, 492 in all, each with a model answer. Free.
More Where the time goes interview questions
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
- You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?
- Explain the optimisations an ML compiler makes, such as kernel fusion, and how they map to the hardware.