Mid-level interview questionAsked at Mistral AI, Amazon · interview write-up
Explain mixture of experts, and what expert parallelism is.
Model answer
An MoE layer swaps one feed-forward block for many experts and a small router that sends each token to the top few, often 2.
Scroll sideways to see the whole diagram
The feed-forward block is replaced by several experts, each its own feed-forward network, and a small router.
Text version
- The feed-forward block is replaced by several experts, each its own feed-forward network, and a small router.
- For each token the router picks the top 2 experts. The other 6 do no work for this token.
- The two outputs are mixed by the router's weights. Total parameters grow with the experts; compute per token stays near that of two.
Shazeer et al. (2017), Outrageously Large Neural Networks; Jiang et al. (2024), Mixtral of Experts (8 experts, top-2 routing).
Parameters grow, but each token uses a slice: Mixtral 8×7B has about 47B parameters and uses about 13B per token. Expert parallelism places experts on different GPUs and ships tokens to them in an all-to-all exchange, so the costs become communication and load balance.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from Mistral AI interview questions, Amazon interview questions.
More MoE & parallelism interview questions
- Compare tensor parallelism and pipeline parallelism.
- Which collective operations does distributed training rely on, and where does each show up?
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
- Design a pretraining pipeline for a mixture-of-experts model that scales.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.