Senior interview questionAsked at Tesla · interview write-up
Compare RNNs, LSTMs and Transformers for sequence modeling, which Transformer variant you used and why, and how model predictive control fits in.
Model answer
RNNs carry one hidden state step by step: cheap per step, but sequential, and gradients vanish over long spans. LSTMs add gates and a cell state so information survives longer. Transformers attend over all steps in parallel, train faster and capture long range, at cost quadratic in length; for streaming prediction I'd pick a causal decoder.
MPC is different: it uses a dynamics model to optimise actions over a short horizon, applies the first and replans. A learned sequence model can be that dynamics model or forecaster, while MPC enforces constraints.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More from Tesla interview questions.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.