Prefill
Processing the whole prompt in one parallel pass before the first output token can be produced.
Interview questions about Prefill
- Users say your summariser 'takes ages to start' on long documents but then writes quickly. Explain what's happening in terms of prefill and decode.
The wait is prefill. Prefill runs the whole prompt in one parallel pass and sets the time to first token; decode then writes one token per step, each costing…
- Design and implement a KV cache for autoregressive decoding in a Transformer: tensor shapes for prefill and per-step decode, per-layer caches, batching, memory limits and beam search.
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline.
- An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?
Not on that evidence. Three runs of the same CPU prefill on our laptop, same model, versions, threads and prompt, gave 219.6, 223.3 and 239.4 ms: 9% apart.
- Request A has a 960-token prompt and 10 output tokens. Request B has a 16-token prompt and 200 output tokens. Same model and machine: which finishes first, and what does that tell you about a chat app that resends the full history every turn?
A. On our GPT-2 small measurements, A is about 220 ms of prefill plus 10 steps of about 10 ms, roughly 0.3 s; B is about 15 ms plus 200 sequential steps of…
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
Time to first token (TTFT), mostly prefill, and total latency, mostly decode.
- Explain how a server like vLLM runs a decoder-only model efficiently.
A decoder-only model runs prefill, then one decode step per token, and decode is memory-bound, so the win is batching many requests.
- Design inference serving for a model with 100 million daily users. Give back-of-the-envelope QPS and memory for weights and caches, and explain how you'd control tail latency.
Assume 100 million users each send 10 requests a day: about 11,600 QPS average, peaks perhaps 2 to 3 times higher.