# Causal mask

> Causal mask: In GPT-style models, the rule that a token can only attend to itself and earlier tokens, never later ones.

Canonical: https://llmdojo.dev/glossary/causal-mask

In GPT-style models, the rule that a token can only attend to itself and earlier tokens, never later ones.

Learn it in the lesson: [The causal mask](https://llmdojo.dev/learn/model-foundations/attention/the-causal-mask) (Model Foundations · Attention: How Tokens Use Their Context).

## Interview questions about Causal mask

- [What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?](https://llmdojo.dev/interview/q/what-does-the-causal-mask-do-and-what-does-it-imply-for)
- [Implement batched multi-head self-attention with a causal mask.](https://llmdojo.dev/interview/q/implement-batched-multi-head-self-attention-with-a-causal-mask)
- [With no starter code, implement a GPT-style transformer decoder from scratch in about 30 minutes.](https://llmdojo.dev/interview/q/with-no-starter-code-implement-a-gpt-style-transformer-decoder-from-scratch)
- [Walk me through Transformer attention: causal and padding masks, cross-attention, why a KV cache helps at inference, and how mixed-precision training works.](https://llmdojo.dev/interview/q/walk-me-through-transformer-attention-causal-and-padding-masks-cross-attention-why)
- [What does the KV cache store, why is using it exact rather than an approximation, and how much does it save?](https://llmdojo.dev/interview/q/what-does-the-kv-cache-store-why-is-using-it-exact-rather)
- [Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.](https://llmdojo.dev/interview/q/explain-transformer-and-mixture-of-experts-fundamentals-including-how-training-differs-from)

## Related
- [Attention](https://llmdojo.dev/glossary/attention)
- [Query, key and value](https://llmdojo.dev/glossary/qkv)
- [Attention head](https://llmdojo.dev/glossary/attention-head)
- [Softmax](https://llmdojo.dev/glossary/softmax)
- [Transformer block](https://llmdojo.dev/glossary/transformer-block)
- [Residual connection](https://llmdojo.dev/glossary/residual-connection)
- [Layer norm](https://llmdojo.dev/glossary/layer-norm)
- [Feed-forward network (MLP)](https://llmdojo.dev/glossary/mlp)
- [Glossary of every term](https://llmdojo.dev/glossary)