Mid-level interview questionAsked at Amazon · interview write-up
Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.
Model answer
A transformer stacks blocks of attention, which mixes tokens, and an MLP, each with a residual. MoE swaps the MLP for many experts and a router picking a few per token.
Scroll sideways to see the whole diagram
The feed-forward block is replaced by several experts, each its own feed-forward network, and a small router.
Text version
- The feed-forward block is replaced by several experts, each its own feed-forward network, and a small router.
- For each token the router picks the top 2 experts. The other 6 do no work for this token.
- The two outputs are mixed by the router's weights. Total parameters grow with the experts; compute per token stays near that of two.
Shazeer et al. (2017), Outrageously Large Neural Networks; Jiang et al. (2024), Mixtral of Experts (8 experts, top-2 routing).
Training runs whole sequences in parallel under the causal mask; MoE adds a load-balancing loss so experts don't collapse. Inference prefills the prompt, then decodes token by token with a KV cache; every expert stays in memory though each token uses few.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from Amazon interview questions.
More MoE & parallelism interview questions
- Explain mixture of experts, and what expert parallelism is.
- Compare tensor parallelism and pipeline parallelism.
- Which collective operations does distributed training rely on, and where does each show up?
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
- Design a pretraining pipeline for a mixture-of-experts model that scales.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.