Mid-level interview questionAsked at Mistral AI, Amazon · interview write-up
Compare tensor parallelism and pipeline parallelism.
Model answer
Tensor parallelism splits each weight matrix across GPUs, so every layer needs a collective step: fast only over very fast links, usually inside one machine. Pipeline parallelism gives each GPU a run of consecutive layers and passes activations along: little communication, but GPUs sit idle in the pipeline bubble unless the batch is split into micro-batches. Large runs combine both with data parallelism.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from Mistral AI interview questions, Amazon interview questions.
More MoE & parallelism interview questions
- Explain mixture of experts, and what expert parallelism is.
- Which collective operations does distributed training rely on, and where does each show up?
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
- Design a pretraining pipeline for a mixture-of-experts model that scales.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.