Attention
Also called self-attention
The mechanism that lets each token take in information from other tokens, weighted by how relevant they score.
Interview questions about Attention
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
Score each token's query against every visible key, divide by √d, softmax, and weight the values.
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
It's a lead to test, not a conclusion. Head 4.3 puts 0.659 on " ladder" from " it", but " did", " not" and " reach" also look mostly at " ladder", and the…
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
Attention compute grows quadratically. Each head scores an n × n grid, so doubling n roughly quadruples that work; the paper puts self-attention per layer at…
- Implement batched multi-head self-attention with a causal mask.
Project once, split into heads, attend in each head, merge. python def mhsa(x, wqkv, wo, h): x: (B, T, d) B, T, d = x.shape q, k, v = (x @ wqkv).split(d, -1)…
- Implement scaled dot-product attention in PyTorch, then tell me where it can go wrong numerically.
- Implement self-attention, give its time and memory complexity, and describe the efficient variants.
- Using only plain Python loops, implement a numerically stable sigmoid, a stable softmax and single-head scaled dot-product attention.
- Quick-fire: explain self-attention, LoRA, Adam versus SGD, and the Vision Transformer.
Self-attention: each token scores its query against every key, softmaxes, and sums the values by weight. LoRA: freeze the weights, train a low-rank update.