Senior interview question
You're sizing GPUs for a chat service. How do you estimate KV-cache memory, and which levers would you pull if it doesn't fit?
Model answer
Multiply it out: 2 (key and value) × layers × heads × head dimension × tokens × bytes per number × batch. GPT-2 small in 32-bit floats needs 72 KiB per token; a full 1,024-token window at batch 64 is 4.50 GiB, nearly ten times the weights. Cache, not weights, can cap context length and concurrent users.
Each factor is a lever: fewer bytes (a lower-precision cache), fewer tokens (a local window, trimmed history), keys and values shared across heads, a smaller batch. I'd model the real mix of context lengths and batch sizes before buying hardware.
Practise the rest
11 Context & KV cache questions, 492 in all, each with a model answer. Free.
More Context & KV cache interview questions
- What is a context window, and why can't GPT-2 small take a 1,025th token even with plenty of memory?
- Users say your summariser 'takes ages to start' on long documents but then writes quickly. Explain what's happening in terms of prefill and decode.
- What does the KV cache store, why is using it exact rather than an approximation, and how much does it save?
- Request A has a 960-token prompt and 10 output tokens. Request B has a 16-token prompt and 200 output tokens. Same model and machine: which finishes first, and what does that tell you about a chat app that resends the full history every turn?
- What are multi-head and grouped-query attention, and what does GQA do to the KV cache?
- Here's a small transformer with a few bugs. Fix them, then add a KV cache for decoding.