Mid-level interview questionAsked at Snap · interview write-up
Compare convolutional networks and Transformers on inductive bias, data and compute needs, and long-range dependencies. Where do hybrids fit?
Model answer
Convolutions build in locality and translation equivariance: each layer sees a small neighbourhood so they learn well from less data, but long-range links need depth or pooling. Self-attention connects every position to every other in one layer with few built-in assumptions, so it needs more data or pretraining, and its cost grows quadratically with sequence length. Hybrids take both: a convolutional stem feeding transformer layers, or convolution stages early and attention later, as in CoAtNet.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More from Snap interview questions.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.