Senior interview questionAsked at Adobe · interview write-up
Walk me through Transformer attention: causal and padding masks, cross-attention, why a KV cache helps at inference, and how mixed-precision training works.
Model answer
Masks set scores to minus infinity before the softmax. The causal mask hides future positions; the padding mask hides pad keys, and pads are excluded from the loss.
Scroll sideways to see the whole diagram
Rows are the token looking (its query), columns the token looked at (its key). Each cell is a score: the query's dot product with the key, divided by √d.
Text version
- Rows are the token looking (its query), columns the token looked at (its key). Each cell is a score: the query's dot product with the key, divided by √d.
- The causal mask: before the softmax, every score where the key comes after the query is set to minus infinity.
- Softmax turns those cells into exactly 0, and the remaining weights in each row still add up to 1.
- Token 1 sees only itself, token 2 sees tokens 1 and 2, and so on. Adding tokens later never changes an earlier row, which is why the KV cache is exact.
The causal mask as taught in lesson 3, on the attention lab's three tokens “The river bank”.
In cross-attention, queries come from the decoder, keys and values from the encoder.
At decode time, past keys and values never change, so the KV cache stores them; each step computes projections for one new token instead of re-running the prefix.
Mixed precision runs matmuls in FP16 or BF16, keeps FP32 master weights and optimizer state, and with FP16 scales the loss so small gradients don't underflow.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More from Adobe interview questions.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.