Mid-level interview question
Users say your summariser 'takes ages to start' on long documents but then writes quickly. Explain what's happening in terms of prefill and decode.
Model answer
The wait is prefill. Prefill runs the whole prompt in one parallel pass and sets the time to first token; decode then writes one token per step, each costing about the same.
On our test machine, GPT-2 small prefilled 16 tokens in 15.3 ms and 960 in 219.6 ms, while a decode step stayed between 8.0 and 9.6 ms. The lever is how much text goes into the prompt.
Practise the rest
11 Context & KV cache questions, 492 in all, each with a model answer. Free.
More Context & KV cache interview questions
- What is a context window, and why can't GPT-2 small take a 1,025th token even with plenty of memory?
- What does the KV cache store, why is using it exact rather than an approximation, and how much does it save?
- You're sizing GPUs for a chat service. How do you estimate KV-cache memory, and which levers would you pull if it doesn't fit?
- Request A has a 960-token prompt and 10 output tokens. Request B has a 16-token prompt and 200 output tokens. Same model and machine: which finishes first, and what does that tell you about a chat app that resends the full history every turn?
- What are multi-head and grouped-query attention, and what does GQA do to the KV cache?
- Here's a small transformer with a few bugs. Fix them, then add a KV cache for decoding.