Senior interview questionAsked at Netflix · interview write-up
What is attention, exactly how are the queries, keys and values computed, and how does a KV cache speed up decoding?
Model answer
Attention builds each token's output as a weighted mix of other tokens. Each hidden vector is multiplied by three learned matrices to give a query, key and value. Scores are query-key dot products divided by the root of the head size, masked causally in a decoder, softmaxed, then used to average the values.
In decoding, past tokens' keys and values never change, so the KV cache stores them. Each step projects only the new token and attends over the cache, instead of reprocessing the prefix, paying in memory that grows with batch times context.
Practise the rest
11 Context & KV cache questions, 492 in all, each with a model answer. Free.
More from Netflix interview questions.
More Context & KV cache interview questions
- What is a context window, and why can't GPT-2 small take a 1,025th token even with plenty of memory?
- Users say your summariser 'takes ages to start' on long documents but then writes quickly. Explain what's happening in terms of prefill and decode.
- What does the KV cache store, why is using it exact rather than an approximation, and how much does it save?
- You're sizing GPUs for a chat service. How do you estimate KV-cache memory, and which levers would you pull if it doesn't fit?
- Request A has a 960-token prompt and 10 output tokens. Request B has a 16-token prompt and 200 output tokens. Same model and machine: which finishes first, and what does that tell you about a chat app that resends the full history every turn?
- What are multi-head and grouped-query attention, and what does GQA do to the KV cache?