Greedy decoding
Always picking the single most likely next token. Deterministic, but prone to repetition.
Interview questions about Greedy decoding
- Greedy decoding on a small model gives fluent text that soon repeats itself word for word. Why does that happen?
It's a feedback loop: once a phrase has appeared, the model rates it more likely to appear again, and greedy always takes the top choice.
- Write beam search that returns each sentence with its probability. How does it differ from greedy decoding?
Keep the k best partial sentences. Each step, extend every one with its likely next tokens, score by the sum of log-probabilities, and keep the top k.
- Your LLM gives different answers to the same input. How would you make inference reproducible, and how many repeated runs would you need for a stable score?
First remove the randomness you control: greedy decoding or a fixed seed, and a pinned model, prompt and parameters.