Senior interview questionAsked at NVIDIA · interview write-up
Explain the optimisations an ML compiler makes, such as kernel fusion, and how they map to the hardware.
Model answer
A compiler turns the model graph into fewer, faster kernels. Kernel fusion matters most: chaining elementwise ops, norms or attention into one kernel keeps intermediates in registers or shared memory instead of round-tripping HBM, and most such ops are memory-bound. Others: constant folding, layout changes to what tensor cores want, lowering to vendor libraries, autotuning tile sizes per GPU, static memory planning, and CUDA graphs against launch overhead. Fitting the hardware means sizing tiles to SM resources and keeping matmuls in tensor-core shapes and precisions.
Practise the rest
10 Where the time goes questions, 492 in all, each with a model answer. Free.
More from NVIDIA interview questions.
More Where the time goes interview questions
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
- You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?