You will be able to
- Explain why a static token embedding cannot represent a word's meaning in context, and what attention adds.
- Describe queries, keys and values as learned projections of each token's vector, and the role each plays.
- Compute scaled dot-product attention step by step: scores = Q·Kᵀ, divide by √d, softmax, weighted sum of values.
- Explain the causal mask in decoder models and predict which attention weights it forces to zero.
- Read real GPT-2 attention patterns (previous-token, same-token, first-token and induction-like heads) and say what attention weights can and cannot show.
- Describe a transformer block (layer norm, multi-head attention, MLP, residual connections) stacked N times.
- Explain why attention's work grows with the square of sequence length, and how that motivates context limits and the KV cache.
Topics
Your review
Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.
Sign in freeSources
Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 3 "Looking Inside Large Language Models", sections "Speeding Up Generation by Caching Keys and Values", "Inside the Transformer Block", "The attention layer at a glance", "How attention is calculated", "Self-attention: Relevance scoring", "Self-attention: Combining information", "More Efficient Attention", "The Transformer Block".
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 2 "Understanding Foundation Models", sections "Transformer architecture", "Attention mechanism", "Transformer block", "Other model architectures"; Ch. 9 "Inference Optimization", section "Attention mechanism optimization".
Attention Is All You Need · Ashish Vaswani et al., 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. NIPS 2017 (arXiv:1706.03762). Used: Section 3.2.1 "Scaled Dot-Product Attention", 3.2.2 "Multi-Head Attention", 3.2.3 "Applications of Attention in our Model", Table 1. Open source
Language Models are Unsupervised Multitask Learners · Alec Radford et al., 2019
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. OpenAI. Used: Section 2.3 "Model". Open source
In-context Learning and Induction Heads · Catherine Olsson et al., 2022
Catherine Olsson, Nelson Elhage, Neel Nanda, et al.. arXiv:2209.11895. Open source
Efficient Streaming Language Models with Attention Sinks · Guangxuan Xiao et al., 2023
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis. ICLR 2024 (arXiv:2309.17453). Open source
Speech and Language Processing, 3rd ed. draft · Daniel Jurafsky et al.
Daniel Jurafsky, James H. Martin. Used: Ch. 7 "Transformers and Pretraining" (further reading; draft of August 19, 2026). Open source