Senior interview questionAsked at TikTok · interview write-up
In a recommender that uses cross-attention between a user's behavior sequence and a candidate item, what exactly are the query, key and value?
Model answer
The candidate item is the query; the user's behaviour sequence supplies the keys and values. Each past item is embedded from its ID, category and context like time gap, then projected into a key and a value; the candidate's embedding is projected to the query.
Softmax over query-key scores weights the history, so the output is a candidate-specific interest vector, the target attention DIN introduced. Cost grows with history length per candidate, so long histories are truncated or filtered first.
Practise the rest
20 Attention questions, 492 in all, each with a model answer. Free.
More from TikTok interview questions.
More Attention interview questions
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
- What's inside one transformer block, and where do most of its parameters sit?
- What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?
- A teammate shows you a GPT-2 head that puts most of a pronoun's attention on the right noun and concludes that the head resolves pronouns. How do you respond?
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
- Implement batched multi-head self-attention with a causal mask.