Mid-level interview questionAsked at Mistral AI · interview write-up
Explain Adam and its bias correction, and why AdamW decouples weight decay.
Model answer
Adam keeps running averages of each gradient and its square, and divides the first by the root of the second, so every weight gets its own step size. Both start at zero, so bias correction divides by one minus beta to the power t.
def adamw(w, g, m, v, t, lr=1e-3, b1=0.9,
b2=0.999, eps=1e-8, wd=0.01):
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g * g
m_hat = m / (1 - b1 ** t)
v_hat = v / (1 - b2 ** t)
w = w - lr * (m_hat / (v_hat ** 0.5 + eps) + wd * w)
return w, m, v
An L2 term in the loss gets divided by that root too, so high-gradient weights decay less. AdamW decouples it: shrink the weights directly, outside the adaptive step.
Practise the rest
35 Training questions, 492 in all, each with a model answer. Free.
More from Mistral AI interview questions.
More Training interview questions
- Implement masked cross-entropy in NumPy that stays numerically stable, with label smoothing and a temperature.
- Build a pipeline that finds math documents in web crawls and scores their quality.
- Explain KL divergence for language models: which direction is which, what happens when supports don't match, and how it's used as a regulariser.
- When would you use mean squared error versus cross-entropy? And what is LoRA?
- Explain policy types in RL and modern policy-gradient methods such as TRPO, PPO and GAE, then compare linear attention and grouped-query attention.
- Predict whether an ad impression is clicked within 24 hours at a 0.7% positive rate, with high-cardinality IDs and delayed labels: how do you build and assess the model?