Senior interview question
You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?
Model answer
Weights plus KV cache per conversation times conversations must fit. Weights take about 26 GB. Each cache is 2 × 40 layers × 5,120 × 2,048 tokens × 2 bytes = 1.68 GB, so the remaining 54 GB holds 32, matching AI Engineering's example. Real servers fit fewer: activations, the framework and fragmentation take memory too.
The cache is linear in tokens and bytes per number, so cap conversation length or store the cache in fewer bytes.
Practise the rest
10 Where the time goes questions, 492 in all, each with a model answer. Free.
More Where the time goes interview questions
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?
- Explain the optimisations an ML compiler makes, such as kernel fusion, and how they map to the hardware.