Top-p (nucleus sampling)
Also called nucleus sampling, top-p
Sampling only among the smallest set of most-likely tokens whose probabilities add up to at least p.
Interview questions about Top-p (nucleus sampling)
- Explain temperature, top-k and top-p. Why would you usually reach for top-p rather than a fixed k?
Temperature divides every logit by T before softmax: low T sharpens, high T flattens, and T near 0 approaches greedy.
- Given logits from a toy language model, implement greedy, top-k and top-p decoding in NumPy.
Greedy takes the argmax. Both samplers sort the probabilities, keep a prefix, renormalise it and sample.
- Your integration tests call a hosted model at temperature 0 and compare the output with an exact string. They fail now and then. What's going on, and what do you change?
The test assumes a determinism a hosted API doesn't promise. Temperature 0 is greedy and uses no random numbers, and our sampling script gave byte-identical…
- Cover the fundamentals: self- versus cross-attention, autoregressive decoding and sampling, where RL fits, and how you evaluate the result.
Self-attention draws queries, keys and values from one sequence; cross-attention takes queries from the decoder and keys and values from another source, like…