Mid-level interview questionAsked at NVIDIA · interview write-up
Walk me through how a model goes from framework code to running on a GPU.
Model answer
In eager PyTorch, each op dispatches to a kernel, often from cuBLAS or cuDNN, and the CPU launches it asynchronously on a CUDA stream. The GPU runs it as thread blocks across its SMs, reading HBM. A compiler like torch.compile instead captures a graph, fuses ops and generates Triton kernels; CUDA graphs replay launches with less CPU overhead. Small ops end up launch- or memory-bound, large matmuls compute-bound on tensor cores.
Practise the rest
10 Where the time goes questions, 492 in all, each with a model answer. Free.
More from NVIDIA interview questions.
More Where the time goes interview questions
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
- You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?