Mid-level interview questionAsked at Netflix · interview write-up
Quick-fire: explain self-attention, LoRA, Adam versus SGD, and the Vision Transformer.
Model answer
Self-attention: each token scores its query against every key, softmaxes, and sums the values by weight.
LoRA: freeze the weights, train a low-rank update.
Adam tracks running averages of each parameter's gradient and squared gradient, giving per-parameter step sizes: the transformer default. SGD uses one global rate and less memory; it often generalises well in vision.
ViT cuts an image into patches, embeds each as a token, and runs a standard transformer encoder.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More from Netflix interview questions.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.