Senior interview questionAsked at Meta · interview write-up
Design a pretraining pipeline for a mixture-of-experts model that scales.
Model answer
Combine parallelisms: expert parallelism spreads experts across GPUs with all-to-all token dispatch, tensor parallelism stays inside a node, and pipeline and data parallelism span nodes. The router needs a load-balancing loss and a capacity factor, or a few experts take most tokens; log per-expert load from day one.
Scroll sideways to see the whole diagram
The feed-forward block is replaced by several experts, each its own feed-forward network, and a small router.
Text version
- The feed-forward block is replaced by several experts, each its own feed-forward network, and a small router.
- For each token the router picks the top 2 experts. The other 6 do no work for this token.
- The two outputs are mixed by the router's weights. Total parameters grow with the experts; compute per token stays near that of two.
Shazeer et al. (2017), Outrageously Large Neural Networks; Jiang et al. (2024), Mixtral of Experts (8 experts, top-2 routing).
All-to-all is the bottleneck, so keep expert groups on the fastest links and overlap dispatch with compute. Add router z-loss for stability, checkpoint often, and validate the data mix on small runs first.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from Meta interview questions.
More MoE & parallelism interview questions
- Explain mixture of experts, and what expert parallelism is.
- Compare tensor parallelism and pipeline parallelism.
- Which collective operations does distributed training rely on, and where does each show up?
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.