Senior interview question
Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
Model answer
Attention compute grows quadratically. Each head scores an n × n grid, so doubling n roughly quadruples that work; the paper puts self-attention per layer at n² · d. GPT-2's 1,024-token window is about a million scores per head. The KV cache grows linearly, with context and batch size, and with learned positions the window is a hard cap.
Fixes trade flexibility for speed and memory: a local attention window, keys and values shared across heads, a cache in fewer bits, or architectures that scale linearly. I'd measure latency and memory at both lengths first.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Implement batched multi-head self-attention with a causal mask.
- A pre-norm transformer block trains badly. What do you check around the norms and residuals?