Senior interview questionAsked at Adobe · interview write-up
Your LLM training and serving is slow. Reason about memory-bound versus compute-bound kernels, the cost of materializing the attention matrix, HFU versus MFU, and which GPU profiling signals you'd read.
Model answer
Compare each kernel's arithmetic intensity with the GPU's ratio of compute to bandwidth. Large matmuls are compute-bound; softmax, norms, elementwise ops and small-batch decode are memory-bound, so fuse them.
Naive attention writes the full sequence-by-sequence score matrix to HBM: quadratic memory and traffic. FlashAttention tiles it through on-chip SRAM instead.
MFU counts only the model's required FLOPs against peak; HFU also counts recomputation, so activation checkpointing raises HFU, not MFU.
Profile: timeline gaps for data loading, launch overhead and communication; per-kernel achieved bandwidth versus tensor-core utilization.
Practise the rest
10 Where the time goes questions, 492 in all, each with a model answer. Free.
More from Adobe interview questions.
More Where the time goes interview questions
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
- You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?