Softmax
Turns a list of scores into probabilities: exponentiate each score, then divide by the total so they sum to 1.
Interview questions about Softmax
- Using only plain Python loops, implement a numerically stable sigmoid, a stable softmax and single-head scaled dot-product attention.
- Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.
AUC is the chance a random positive outscores a random negative, so compute it from average ranks, which counts ties as half.
- Implement the forward and backward passes of softmax with cross-entropy from scratch, then write a training loop for a linear classifier.
- Training a retrieval model over a catalogue of millions of items, you use sampled softmax. Why does it need a correction, and how do you choose the classification loss?
A full softmax over millions of items is too costly, so each positive competes against sampled negatives.
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
Score each token's query against every visible key, divide by √d, softmax, and weight the values.
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
It hides later tokens: before softmax, every score whose key comes after the query becomes minus infinity, so those weights are exactly 0 and the rest still…
- Implement batched multi-head self-attention with a causal mask.
Project once, split into heads, attend in each head, merge. python def mhsa(x, wqkv, wo, h): x: (B, T, d) B, T, d = x.shape q, k, v = (x @ wqkv).split(d, -1)…
- Implement scaled dot-product attention in PyTorch, then tell me where it can go wrong numerically.