Mid-level interview questionAsked at Applied Intuition · interview write-up
This notebook trains a minimal GPT but the loss won't drop. Find and fix the bugs, starting with how the causal and padding masks are applied.
Model answer
T = x.size(1)
causal = torch.ones(T, T, dtype=torch.bool,
device=x.device).tril()
keep = causal[None, None] & pad[:, None, None, :]
att = att.masked_fill(~keep, float("-inf"))
att = att.softmax(-1)
# later: predict x[t+1] from tokens up to t
logits = model(x[:, :-1])
loss = F.cross_entropy(logits.transpose(1, 2),
x[:, 1:], ignore_index=PAD)
Classic bugs: triu where you meant tril; masking after the softmax, or with zero instead of minus infinity; a padding mask on queries instead of keys; a fully masked row turning into NaN; targets not shifted by one; pads counted in the loss. Then the loop: zero_grad, train mode, learning rate.
Practise the rest
35 Training questions, 492 in all, each with a model answer. Free.
More from Applied Intuition interview questions.
More Training interview questions
- Implement masked cross-entropy in NumPy that stays numerically stable, with label smoothing and a temperature.
- Build a pipeline that finds math documents in web crawls and scores their quality.
- Explain KL divergence for language models: which direction is which, what happens when supports don't match, and how it's used as a regulariser.
- When would you use mean squared error versus cross-entropy? And what is LoRA?
- Explain Adam and its bias correction, and why AdamW decouples weight decay.
- Explain policy types in RL and modern policy-gradient methods such as TRPO, PPO and GAE, then compare linear attention and grouped-query attention.