Transformer block
Also called transformer blocks
The repeated unit of a transformer: attention and a feed-forward network, each wrapped in a layer norm and a residual connection.
Interview questions about Transformer block
- What's inside one transformer block, and where do most of its parameters sit?
Two sub-blocks, each layer norm, then the sublayer, then a residual add: first multi-head attention, then an MLP.
- A pre-norm transformer block trains badly. What do you check around the norms and residuals?
Pre-norm should read x + attn(norm(x)), then x + mlp(norm(x)): normalise each sublayer's input and add its output to the un-normalised stream.
- Implement scaled dot-product multi-head attention and a Transformer block from scratch, including the backward pass for attention, without autograd.
Have the forward pass cache what the backward pass needs: the softmax probabilities P.