Junior interview question
What's inside one transformer block, and where do most of its parameters sit?
Model answer
Two sub-blocks, each layer norm, then the sublayer, then a residual add: first multi-head attention, then an MLP.
Scroll sideways to see the whole diagram
Each token enters as its embedding plus a position signal, so the model knows the order of the tokens.
Text version
- Each token enters as its embedding plus a position signal, so the model knows the order of the tokens.
- Attention: every token looks at itself and the tokens before it, and pulls in information from them. (A layer norm first rescales each vector to keep the numbers stable.)
- The attention result is added back onto the token's own vector: a residual connection. The block refines each vector instead of replacing it.
- A feed-forward network then processes each token's vector on its own: attention mixes information between tokens, the MLP transforms it within a token.
- Another residual add, and one block is done. The model stacks many identical blocks with different learned weights; each block's output feeds the next.
- After the last block, each position's final vector is turned into a score for every token in the vocabulary. Turning those scores into text is the next lesson.
Structure of a pre-norm decoder block as used by GPT-2: layer norm before attention and before the feed-forward network, each followed by a residual add.
Attention moves information between tokens; the MLP transforms each token on its own and holds most parameters. In GPT-2 small's first block, the MLP has 4,722,432 against attention's 2,362,368, and the model stacks 12 blocks.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.
- A pre-norm transformer block trains badly. What do you check around the norms and residuals?