Senior interview questionAsked at Anthropic · interview write-up
Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
Model answer
Count every live tensor.
# Y = X @ W; X (m, k), W (k, n); bf16 = 2 bytes
fwd = 2 * (m * k + k * n + m * n)
# m = k = n = 65536: 8 GiB each, 24 GiB total
Training adds gradients, and mixed-precision Adam needs about 16 bytes per parameter with its FP32 copies. Tensor parallelism splits W itself, so each GPU holds a slice and every step needs an all-reduce or all-gather, best over NVLink. Pipeline parallelism gives whole layers to stages and passes only activations, fine across nodes but with bubbles. It can't split a single matmul too big for one GPU; tensor parallelism can.
Practise the rest
10 MoE & parallelism questions, 492 in all, each with a model answer. Free.
More from Anthropic interview questions.
More MoE & parallelism interview questions
- Explain mixture of experts, and what expert parallelism is.
- Compare tensor parallelism and pipeline parallelism.
- Which collective operations does distributed training rely on, and where does each show up?
- Design a pretraining pipeline for a mixture-of-experts model that scales.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.