Mid-level interview questionAsked at LinkedIn · interview write-up
Compare the common activation functions, explain how you choose a loss for different problems, and walk me through how Adam works and its hyperparameters.
Model answer
Sigmoid and tanh saturate, so gradients vanish; ReLU is cheap but units can die; GELU and SiLU are smooth and standard in transformers.
The loss follows the output distribution: MSE for Gaussian-ish regression, Huber against outliers, sigmoid plus binary cross-entropy, softmax plus cross-entropy.
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g * g
m_hat = m / (1 - b1 ** t)
v_hat = v / (1 - b2 ** t)
w -= lr * m_hat / (np.sqrt(v_hat) + eps)
Adam keeps momentum and a per-parameter scale, bias-corrected. Defaults: beta1 0.9, beta2 0.999, eps 1e-8; tune the learning rate first.
Practise the rest
35 Training questions, 492 in all, each with a model answer. Free.
More from LinkedIn interview questions.
More Training interview questions
- Implement masked cross-entropy in NumPy that stays numerically stable, with label smoothing and a temperature.
- Build a pipeline that finds math documents in web crawls and scores their quality.
- Explain KL divergence for language models: which direction is which, what happens when supports don't match, and how it's used as a regulariser.
- When would you use mean squared error versus cross-entropy? And what is LoRA?
- Explain Adam and its bias correction, and why AdamW decouples weight decay.
- Explain policy types in RL and modern policy-gradient methods such as TRPO, PPO and GAE, then compare linear attention and grouped-query attention.