Lesson 5 · 8 topics · 22 min · beginner

Next-Token Prediction and Sampling

You will be able to

Explain how the LM head turns the last position's vector into one logit per vocabulary token, and how softmax turns logits into a probability distribution.
Describe autoregressive generation as a loop: pick a token, append it, run the model again.
Explain greedy decoding and why it can lock into repetition loops.
Predict how temperature reshapes a distribution (low T sharpens, high T flattens, T near 0 approaches greedy).
Compare top-k and top-p (nucleus) sampling and explain why top-p adapts to the shape of the distribution.
Explain what a seed does, and why fixed settings make outputs repeatable on one setup but are not a guarantee across hardware or hosted APIs.
Choose sampling settings for a task (low temperature for one-correct-answer tasks, higher for variety) as guidance to test, not a rule.
Use logprobs as a confidence and debugging signal while recognising that probability is not calibrated correctness.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 2 "Understanding Foundation Models", section "Sampling": subsections "Sampling Fundamentals", "Sampling Strategies", "Temperature", "Top-k", "Top-p", "Test Time Compute", "The Probabilistic Nature of AI" ("Inconsistency"); Ch. 4 "Evaluate AI Systems", section "Select evaluation methods".
Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 3 "Looking Inside Large Language Models", section "Choosing a Single Token from the Probability Distribution (Sampling/Decoding)"; Ch. 6 "Prompt Engineering", section "Controlling Model Output" (subsections "Temperature", "top_p").
The Curious Case of Neural Text Degeneration · Ari Holtzman et al., 2020
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi. ICLR 2020 (arXiv:1904.09751). Used: Abstract; Section 3.1 "Nucleus Sampling", 3.2 "Top-k Sampling", 3.3 "Sampling with Temperature"; Figures 4 and 5. Open source
GPT-4 Technical Report · OpenAI, 2023
OpenAI. arXiv:2303.08774. Used: Section "Limitations", Figure 8 (calibration). Open source

You will be able to

Explain how the LM head turns the last position's vector into one logit per vocabulary token, and how softmax turns logits into a probability distribution.

Describe autoregressive generation as a loop: pick a token, append it, run the model again.

Explain greedy decoding and why it can lock into repetition loops.

Predict how temperature reshapes a distribution (low T sharpens, high T flattens, T near 0 approaches greedy).

Compare top-k and top-p (nucleus) sampling and explain why top-p adapts to the shape of the distribution.

Explain what a seed does, and why fixed settings make outputs repeatable on one setup but are not a guarantee across hardware or hosted APIs.

Choose sampling settings for a task (low temperature for one-correct-answer tasks, higher for variety) as guidance to test, not a rule.

Use logprobs as a confidence and debugging signal while recognising that probability is not calibrated correctness.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 2 "Understanding Foundation Models", section "Sampling": subsections "Sampling Fundamentals", "Sampling Strategies", "Temperature", "Top-k", "Top-p", "Test Time Compute", "The Probabilistic Nature of AI" ("Inconsistency"); Ch. 4 "Evaluate AI Systems", section "Select evaluation methods".

Hands-On Large Language Models · Jay Alammar et al., 2024

Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 3 "Looking Inside Large Language Models", section "Choosing a Single Token from the Probability Distribution (Sampling/Decoding)"; Ch. 6 "Prompt Engineering", section "Controlling Model Output" (subsections "Temperature", "top_p").

The Curious Case of Neural Text Degeneration · Ari Holtzman et al., 2020

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi. ICLR 2020 (arXiv:1904.09751). Used: Abstract; Section 3.1 "Nucleus Sampling", 3.2 "Top-k Sampling", 3.3 "Sampling with Temperature"; Figures 4 and 5. Open source

GPT-4 Technical Report · OpenAI, 2023

OpenAI. arXiv:2303.08774. Used: Section "Limitations", Figure 8 (calibration). Open source