Senior interview questionAsked at Mistral AI · interview write-up
A pre-norm transformer block trains badly. What do you check around the norms and residuals?
Model answer
Pre-norm should read x + attn(norm(x)), then x + mlp(norm(x)): normalise each sublayer's input and add its output to the un-normalised stream.
Scroll sideways to see the whole diagram
Each token enters as its embedding plus a position signal, so the model knows the order of the tokens.
Text version
- Each token enters as its embedding plus a position signal, so the model knows the order of the tokens.
- Attention: every token looks at itself and the tokens before it, and pulls in information from them. (A layer norm first rescales each vector to keep the numbers stable.)
- The attention result is added back onto the token's own vector: a residual connection. The block refines each vector instead of replacing it.
- A feed-forward network then processes each token's vector on its own: attention mixes information between tokens, the MLP transforms it within a token.
- Another residual add, and one block is done. The model stacks many identical blocks with different learned weights; each block's output feeds the next.
- After the last block, each position's final vector is turned into a score for every token in the vocabulary. Turning those scores into text is the next lesson.
Structure of a pre-norm decoder block as used by GPT-2: layer norm before attention and before the feed-forward network, each followed by a residual add.
Common bugs: adding to the normed tensor, so the residual path stops being an identity; normalising the sublayer's output instead of its input; one norm shared where two are meant; and no final norm before the output layer. A quick test: zero both output projections, and the block must return x exactly.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More from Mistral AI interview questions.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.