You will be able to
- Explain the context window as a hard limit on positions (GPT-2 small: n_positions = 1,024) shared by the prompt and the generated tokens, and what happens when an input exceeds it.
- Explain why a transformer needs position information and describe learned absolute position embeddings, reading GPT-2's measured position-vector similarities; name RoPE as an alternative.
- Distinguish prefill (one parallel pass over the prompt) from decode (one token at a time), and connect them to time to first token and time per output token.
- Explain how the KV cache stores each layer's keys and values for earlier tokens so each decode step computes only the new token's, and why this is exact.
- Compute KV-cache memory as 2 × layers × heads × head dimension × tokens × bytes (× batch) for GPT-2 small, and predict how each factor changes it.
- Estimate how prompt length and output length affect a request's latency from measured prefill and decode times, comparing ratios rather than absolute times.
Topics
Your review
Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.
Sign in freeSources
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 2 "Understanding Foundation Models", section "Transformer architecture" (prefill and decode) and subsection "Transformer block" (embedding module, position indices); Ch. 4 "Evaluate AI Systems", section "Cost and Latency"; Ch. 9 "Inference Optimization", sections "Inference Overview" ("Computational bottlenecks"), "Inference Performance Metrics" ("Latency, TTFT, and TPOT"), "Attention mechanism optimization" (sidebar "Calculating the KV Cache Size").
Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 3 "Looking Inside Large Language Models", sections "Speeding Up Generation by Caching Keys and Values", "Positional Embeddings (RoPE)".
Attention Is All You Need · Ashish Vaswani et al., 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. NeurIPS 2017 (arXiv:1706.03762). Used: Section 3.5 "Positional Encoding". Open source
Language Models are Unsupervised Multitask Learners · Alec Radford et al., 2019
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. OpenAI. Used: Section 2.3 "Model". Open source
RoFormer: Enhanced Transformer with Rotary Position Embedding · Jianlin Su et al., 2021
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu. arXiv:2104.09864. Used: Abstract. Open source
GPT-2 model configuration (config.json) · OpenAI (Hugging Face model repository openai-community/gpt2)
OpenAI (Hugging Face model repository openai-community/gpt2). Used: n_positions, n_ctx, n_layer, n_head, n_embd. Open source