Senior interview questionAsked at Snap · interview write-up
Compare batch normalization and layer normalization: which axes does each normalize over, how do they differ between training and inference, and how would you debug a normalization bug?
Model answer
BatchNorm normalizes each feature across the batch. LayerNorm normalizes each example across its own features. So BatchNorm depends on the batch: training uses batch statistics and updates running averages, inference uses those stored running statistics. LayerNorm computes the same thing in both modes, which is why transformers and small or variable batches use it.
Bugs usually show up as a train-eval gap: a missing model.eval(), tiny or unshuffled batches, stale running statistics after fine-tuning, or the wrong axis. Compare train and eval outputs on one batch and inspect the running statistics.
Practise the rest
35 Training questions, 492 in all, each with a model answer. Free.
More from Snap interview questions.
More Training interview questions
- Implement masked cross-entropy in NumPy that stays numerically stable, with label smoothing and a temperature.
- Build a pipeline that finds math documents in web crawls and scores their quality.
- Explain KL divergence for language models: which direction is which, what happens when supports don't match, and how it's used as a regulariser.
- When would you use mean squared error versus cross-entropy? And what is LoRA?
- Explain Adam and its bias correction, and why AdamW decouples weight decay.
- Explain policy types in RL and modern policy-gradient methods such as TRPO, PPO and GAE, then compare linear attention and grouped-query attention.