Causal mask
In GPT-style models, the rule that a token can only attend to itself and earlier tokens, never later ones.
Interview questions about Causal mask
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
It hides later tokens: before softmax, every score whose key comes after the query becomes minus infinity, so those weights are exactly 0 and the rest still…
- Implement batched multi-head self-attention with a causal mask.
Project once, split into heads, attend in each head, merge. python def mhsa(x, wqkv, wo, h): x: (B, T, d) B, T, d = x.shape q, k, v = (x @ wqkv).split(d, -1)…
- With no starter code, implement a GPT-style transformer decoder from scratch in about 30 minutes.
- Walk me through Transformer attention: causal and padding masks, cross-attention, why a KV cache helps at inference, and how mixed-precision training works.
Masks set scores to minus infinity before the softmax. The causal mask hides future positions; the padding mask hides pad keys, and pads are excluded from the…
- What does the KV cache store, why is using it exact rather than an approximation, and how much does it save?
It stores every processed token's keys and values, per layer and head, so each decode step computes only the new token's.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.
A transformer stacks blocks of attention, which mixes tokens, and an MLP, each with a residual.