Mid-level interview questionAsked at Scale AI · interview write-up
Cover the fundamentals: self- versus cross-attention, autoregressive decoding and sampling, where RL fits, and how you evaluate the result.
Model answer
Self-attention draws queries, keys and values from one sequence; cross-attention takes queries from the decoder and keys and values from another source, like an encoder's output.
Scroll sideways to see the whole diagram
The model reads the whole prompt: every position goes through all the transformer blocks.
Text version
- The model reads the whole prompt: every position goes through all the transformer blocks.
- To predict what comes next, only the vector at the last position is used.
- The LM head turns that vector into one score per vocabulary token: a logit. Higher means more likely next.
- Softmax turns the scores into probabilities. Temperature divides the scores first: below 1 sharpens the distribution, above 1 flattens it.
- One token is chosen: greedy takes the most likely; sampling draws at random, weighted by probability, optionally only among the top-k or top-p candidates.
- The chosen token is appended and the whole loop runs again. One loop per generated token, until a stop token or a length limit.
The standard autoregressive decoding loop of a GPT-style model. The vocabulary size is GPT-2's 50,257 tokens.
Decoding is a loop: score the next token, pick one greedily or by temperature and top-p, append, repeat, with a KV cache. RL then tunes those sampled outputs against a reward, held near a reference model. Evaluate with perplexity, task benchmarks and human or judge preferences.
Practise the rest
9 Sampling questions, 492 in all, each with a model answer. Free.
More from Scale AI interview questions.
More Sampling interview questions
- What are logits, and how does a model go from them to a whole sentence of output?
- Explain temperature, top-k and top-p. Why would you usually reach for top-p rather than a fixed k?
- Greedy decoding on a small model gives fluent text that soon repeats itself word for word. Why does that happen?
- Your integration tests call a hosted model at temperature 0 and compare the output with an exact string. They fail now and then. What's going on, and what do you change?
- A PM wants to show users a 'confidence' percentage computed from the answer's token probabilities. What's your position, and what would you build instead?
- Write beam search that returns each sentence with its probability. How does it differ from greedy decoding?