Senior interview questionAsked at Snap · interview write-up
What happens if every weight in a neural network starts at zero, and how does the answer change for linear and logistic regression?
Model answer
With hidden layers, every unit in a layer computes the same output and gets the same gradient, so they stay identical forever: the layer acts like one unit. Worse, with all weights zero the gradients reaching earlier layers are zero too, so they don't move at all.
Linear and logistic regression are fine. There are no hidden units to be symmetric, the loss is convex, and gradient descent from zero reaches the same optimum as from anywhere. Networks therefore use random initialization scaled to keep activation variance steady, such as Xavier or He initialization.
Practise the rest
35 Training questions, 492 in all, each with a model answer. Free.
More from Snap interview questions.
More Training interview questions
- Implement masked cross-entropy in NumPy that stays numerically stable, with label smoothing and a temperature.
- Build a pipeline that finds math documents in web crawls and scores their quality.
- Explain KL divergence for language models: which direction is which, what happens when supports don't match, and how it's used as a regulariser.
- When would you use mean squared error versus cross-entropy? And what is LoRA?
- Explain Adam and its bias correction, and why AdamW decouples weight decay.
- Explain policy types in RL and modern policy-gradient methods such as TRPO, PPO and GAE, then compare linear attention and grouped-query attention.