# Attention

> Attention: The mechanism that lets each token take in information from other tokens, weighted by how relevant they score.

Canonical: https://llmdojo.dev/glossary/attention

Also called self-attention.

The mechanism that lets each token take in information from other tokens, weighted by how relevant they score.

Learn it in the lesson: [One vector per token is not enough](https://llmdojo.dev/learn/model-foundations/attention/one-vector-is-not-enough) (Model Foundations · Attention: How Tokens Use Their Context).

## Interview questions about Attention

- [Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?](https://llmdojo.dev/interview/q/walk-me-through-scaled-dot-product-attention-what-are-queries-keys-and)
- [A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?](https://llmdojo.dev/interview/q/a-teammate-shows-you-a-gpt-2-head-that-puts-most-of)
- [Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?](https://llmdojo.dev/interview/q/product-wants-to-double-the-context-you-support-from-1-000-to)
- [Implement batched multi-head self-attention with a causal mask.](https://llmdojo.dev/interview/q/implement-batched-multi-head-self-attention-with-a-causal-mask)
- [Implement scaled dot-product attention in PyTorch, then tell me where it can go wrong numerically.](https://llmdojo.dev/interview/q/implement-scaled-dot-product-attention-in-pytorch-then-tell-me-where-it)
- [Implement self-attention, give its time and memory complexity, and describe the efficient variants.](https://llmdojo.dev/interview/q/implement-self-attention-give-its-time-and-memory-complexity-and-describe-the)
- [Using only plain Python loops, implement a numerically stable sigmoid, a stable softmax and single-head scaled dot-product attention.](https://llmdojo.dev/interview/q/using-only-plain-python-loops-implement-a-numerically-stable-sigmoid-a-stable)
- [Quick-fire: explain self-attention, LoRA, Adam versus SGD, and the Vision Transformer.](https://llmdojo.dev/interview/q/quick-fire-explain-self-attention-lora-adam-versus-sgd-and-the-vision)

## Related
- [Query, key and value](https://llmdojo.dev/glossary/qkv)
- [Attention head](https://llmdojo.dev/glossary/attention-head)
- [Causal mask](https://llmdojo.dev/glossary/causal-mask)
- [Softmax](https://llmdojo.dev/glossary/softmax)
- [Transformer block](https://llmdojo.dev/glossary/transformer-block)
- [Residual connection](https://llmdojo.dev/glossary/residual-connection)
- [Layer norm](https://llmdojo.dev/glossary/layer-norm)
- [Feed-forward network (MLP)](https://llmdojo.dev/glossary/mlp)
- [Glossary of every term](https://llmdojo.dev/glossary)