Mid-level interview questionAsked at NVIDIA · interview write-up
Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
Model answer
Quantization stores weights or activations in fewer bits. Pruning removes weights, and only structured sparsity such as 2:4 speeds up GPUs. Distillation trains a small student on a big teacher's outputs. Fusion merges kernels to cut memory traffic.
Tensor parallelism shards each layer's matrices, with all-reduces every layer, so it wants NVLink. Pipeline parallelism assigns layers to stages, sends only activations, and suffers bubbles that micro-batches shrink.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from NVIDIA interview questions.
More MoE & parallelism interview questions
- Explain mixture of experts, and what expert parallelism is.
- Compare tensor parallelism and pipeline parallelism.
- Which collective operations does distributed training rely on, and where does each show up?
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
- Design a pretraining pipeline for a mixture-of-experts model that scales.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.