Layer norm
Also called layer normalization
Rescales each token's vector to a standard spread before a sub-layer, which keeps training stable.
Interview questions about Layer norm
- Compare batch normalization and layer normalization: which axes does each normalize over, how do they differ between training and inference, and how would you debug a normalization bug?
BatchNorm normalizes each feature across the batch. LayerNorm normalizes each example across its own features.
- What's inside one transformer block, and where do most of its parameters sit?
Two sub-blocks, each layer norm, then the sublayer, then a residual add: first multi-head attention, then an MLP.