Mid-level interview questionAsked at OpenAI · interview write-up
Implement masked cross-entropy in NumPy that stays numerically stable, with label smoothing and a temperature.
Model answer
Work in log-probs and subtract the row max before the log-sum-exp, so nothing overflows.
import numpy as np
def masked_ce(logits, y, mask, eps=0.1, T=1.0):
# logits (N, V), y (N,) ints, mask (N,) 0/1
z = logits / T
z = z - z.max(-1, keepdims=True)
logp = z - np.log(np.exp(z).sum(-1, keepdims=True))
y = np.where(mask > 0, y, 0) # ignored ids safe
nll = -logp[np.arange(len(y)), y]
loss = (1 - eps) * nll - eps * logp.mean(-1)
m = mask.astype(loss.dtype)
return (loss * m).sum() / max(m.sum(), 1.0)
Smoothing mixes the one-hot target with a uniform one, which is a weighted sum of the NLL and the mean negative log-prob. Interviewers check you average over unmasked tokens only, guard an all-masked batch, and never index with ignore labels like -100.
Practise the rest
35 Training questions, 492 in all, each with a model answer. Free.
More from OpenAI interview questions.
More Training interview questions
- Build a pipeline that finds math documents in web crawls and scores their quality.
- Explain KL divergence for language models: which direction is which, what happens when supports don't match, and how it's used as a regulariser.
- When would you use mean squared error versus cross-entropy? And what is LoRA?
- Explain Adam and its bias correction, and why AdamW decouples weight decay.
- Explain policy types in RL and modern policy-gradient methods such as TRPO, PPO and GAE, then compare linear attention and grouped-query attention.
- Predict whether an ad impression is clicked within 24 hours at a 0.7% positive rate, with high-cardinality IDs and delayed labels: how do you build and assess the model?