Decode
Also called decode steps, decode step, decode phase
The sequential phase of generation: one forward pass per new token.
Interview questions about Decode
- Implement a byte-pair encoding tokenizer with train, encode and decode, where training keeps merging the most frequent adjacent pair until counts fall below a threshold.
Work on bytes so any text encodes. Training merges the most frequent adjacent pair into a new ID until its count falls below the threshold.
- Users say your summariser 'takes ages to start' on long documents but then writes quickly. Explain what's happening in terms of prefill and decode.
The wait is prefill. Prefill runs the whole prompt in one parallel pass and sets the time to first token; decode then writes one token per step, each costing…
- Design and implement a KV cache for autoregressive decoding in a Transformer: tensor shapes for prefill and per-step decode, per-layer caches, batching, memory limits and beam search.
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
Compute-bound work is limited by arithmetic; bandwidth-bound work by how fast data moves from memory.
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline.
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
Each step reads the weights once and uses them for every sequence; only each sequence's KV cache is extra.
- A colleague says INT4 weights will make decode eight times faster because they're eight times smaller. How do you respond?
It's a ceiling, not a forecast. Batch-1 decode is bandwidth-bound, so time should track bytes read, but the KV cache doesn't shrink.
- Walk me through Transformer attention: causal and padding masks, cross-attention, why a KV cache helps at inference, and how mixed-precision training works.
Masks set scores to minus infinity before the softmax. The causal mask hides future positions; the padding mask hides pad keys, and pads are excluded from the…