# Softmax

> Softmax: Turns a list of scores into probabilities: exponentiate each score, then divide by the total so they sum to 1.

Canonical: https://llmdojo.dev/glossary/softmax

Turns a list of scores into probabilities: exponentiate each score, then divide by the total so they sum to 1.

Learn it in the lesson: [Scaled dot-product attention](https://llmdojo.dev/learn/model-foundations/attention/scaled-dot-product) (Model Foundations · Attention: How Tokens Use Their Context).

## Interview questions about Softmax

- [Using only plain Python loops, implement a numerically stable sigmoid, a stable softmax and single-head scaled dot-product attention.](https://llmdojo.dev/interview/q/using-only-plain-python-loops-implement-a-numerically-stable-sigmoid-a-stable)
- [Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.](https://llmdojo.dev/interview/q/implement-auc-roc-with-correct-tie-handling-a-numerically-stable-softmax-and)
- [Implement the forward and backward passes of softmax with cross-entropy from scratch, then write a training loop for a linear classifier.](https://llmdojo.dev/interview/q/implement-the-forward-and-backward-passes-of-softmax-with-cross-entropy-from)
- [Training a retrieval model over a catalogue of millions of items, you use sampled softmax. Why does it need a correction, and how do you choose the classification loss?](https://llmdojo.dev/interview/q/training-a-retrieval-model-over-a-catalogue-of-millions-of-items-you)
- [Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?](https://llmdojo.dev/interview/q/walk-me-through-scaled-dot-product-attention-what-are-queries-keys-and)
- [What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?](https://llmdojo.dev/interview/q/what-does-the-causal-mask-do-and-what-does-it-imply-for)
- [Implement batched multi-head self-attention with a causal mask.](https://llmdojo.dev/interview/q/implement-batched-multi-head-self-attention-with-a-causal-mask)
- [Implement scaled dot-product attention in PyTorch, then tell me where it can go wrong numerically.](https://llmdojo.dev/interview/q/implement-scaled-dot-product-attention-in-pytorch-then-tell-me-where-it)

## Related
- [Attention](https://llmdojo.dev/glossary/attention)
- [Query, key and value](https://llmdojo.dev/glossary/qkv)
- [Attention head](https://llmdojo.dev/glossary/attention-head)
- [Causal mask](https://llmdojo.dev/glossary/causal-mask)
- [Transformer block](https://llmdojo.dev/glossary/transformer-block)
- [Residual connection](https://llmdojo.dev/glossary/residual-connection)
- [Layer norm](https://llmdojo.dev/glossary/layer-norm)
- [Feed-forward network (MLP)](https://llmdojo.dev/glossary/mlp)
- [Glossary of every term](https://llmdojo.dev/glossary)