Senior interview questionAsked at Amazon · interview write-up
Which collective operations does distributed training rely on, and where does each show up?
Model answer
All-reduce sums gradients across data-parallel replicas, and partial results in tensor parallelism. All-gather and reduce-scatter are its two halves: sharded training such as ZeRO or FSDP gathers each layer's weights just before use and reduce-scatters its gradients after. All-to-all routes tokens to experts in MoE. Each costs bytes over links, so a good layout puts the chattiest collectives on the fastest links.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from Amazon interview questions.
More MoE & parallelism interview questions
- Explain mixture of experts, and what expert parallelism is.
- Compare tensor parallelism and pipeline parallelism.
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
- Design a pretraining pipeline for a mixture-of-experts model that scales.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.