Mid-level interview questionAsked at Cohere · YouTube
Explain temperature, top-k and top-p. Why would you usually reach for top-p rather than a fixed k?
Model answer
Temperature divides every logit by T before softmax: low T sharpens, high T flattens, and T near 0 approaches greedy. Top-k keeps a fixed number of top tokens; top-p keeps the smallest set whose probabilities reach p, so the cut adapts to the distribution's shape.
At top-p 0.9, GPT-2 kept 1 token after "Thank you very" but 417 for an open-ended prompt. No single k suits both: Holtzman et al.'s argument.
Practise the rest
9 Sampling questions, 492 in all, each with a model answer. Free.
More from Cohere interview questions.
More Sampling interview questions
- What are logits, and how does a model go from them to a whole sentence of output?
- Greedy decoding on a small model gives fluent text that soon repeats itself word for word. Why does that happen?
- Your integration tests call a hosted model at temperature 0 and compare the output with an exact string. They fail now and then. What's going on, and what do you change?
- A PM wants to show users a 'confidence' percentage computed from the answer's token probabilities. What's your position, and what would you build instead?
- Write beam search that returns each sentence with its probability. How does it differ from greedy decoding?
- Build a bigram next-word predictor that samples the next word in proportion to how often it followed.