# Next-token prediction and sampling interview questions

> 9 next-token prediction and sampling interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/sampling
Last updated: 2026-10-23

9 questions: 3 junior, 4 mid-level and 2 senior. Taught in [Next-Token Prediction and Sampling](https://llmdojo.dev/learn/model-foundations/sampling).

## [What are logits, and how does a model go from them to a whole sentence of output?](https://llmdojo.dev/interview/q/what-are-logits-and-how-does-a-model-go-from-them-to)

Level: Junior

Logits are the **raw scores** the LM head gives every vocabulary token, **50,257** for GPT-2, from the last position's vector. Softmax turns them into probabilities: after "Thank you very", " much" got 0.992.

**Diagram: How a model writes, one token at a time**

1. The model reads the whole prompt: every position goes through all the transformer blocks.
2. To predict what comes next, only the vector at the last position is used.
3. The LM head turns that vector into one score per vocabulary token: a logit. Higher means more likely next.
4. Softmax turns the scores into probabilities. Temperature divides the scores first: below 1 sharpens the distribution, above 1 flattens it.
5. One token is chosen: greedy takes the most likely; sampling draws at random, weighted by probability, optionally only among the top-k or top-p candidates.
6. The chosen token is appended and the whole loop runs again. One loop per generated token, until a stop token or a length limit.

The standard autoregressive decoding loop of a GPT-style model. The vocabulary size is GPT-2's 50,257 tokens.

For a sentence, loop: pick a token, append it, **run the model again**. So 25 new tokens means 25 passes.

## [Explain temperature, top-k and top-p. Why would you usually reach for top-p rather than a fixed k?](https://llmdojo.dev/interview/q/explain-temperature-top-k-and-top-p-why-would-you-usually-reach)

Level: Mid-level

Temperature **divides every logit by T** before softmax: low T sharpens, high T flattens, and T near 0 approaches greedy. Top-k keeps a fixed number of top tokens; top-p keeps the smallest set whose probabilities reach p, so **the cut adapts** to the distribution's shape.

At top-p 0.9, GPT-2 kept 1 token after "Thank you very" but 417 for an open-ended prompt. **No single k suits both**: Holtzman et al.'s argument.

## [Greedy decoding on a small model gives fluent text that soon repeats itself word for word. Why does that happen?](https://llmdojo.dev/interview/q/greedy-decoding-on-a-small-model-gives-fluent-text-that-soon-repeats)

Level: Mid-level

It's a **feedback loop**: once a phrase has appeared, the model rates it **more likely to appear again**, and greedy always takes the top choice. Continuing "I opened the box and found", GPT-2 small picked " amount" at 0.027 the first time and **0.988 on the repeat**.

Holtzman et al. report the same for most phrases they tested: decoding that maximizes probability tends to give bland or looping text.

## [Your integration tests call a hosted model at temperature 0 and compare the output with an exact string. They fail now and then. What's going on, and what do you change?](https://llmdojo.dev/interview/q/your-integration-tests-call-a-hosted-model-at-temperature-0-and-compare)

Level: Senior

The test assumes a determinism a hosted API doesn't promise. Temperature 0 is greedy and uses no random numbers, and our sampling script gave byte-identical output across two runs, but on one machine. *AI Engineering* notes that even with temperature, top-p, top-k and seed fixed, **hardware can change the output**, and with an API the hardware is the provider's choice.

So **stop asserting exact text** from live calls: cache known answers where you need exact strings, or parse the output and compare fields, labels or structure. A seed won't fix it.

## [A PM wants to show users a 'confidence' percentage computed from the answer's token probabilities. What's your position, and what would you build instead?](https://llmdojo.dev/interview/q/a-pm-wants-to-show-users-a-confidence-percentage-computed-from-the)

Level: Senior

Against it: a token probability says **how typical the text is** after the prompt, **not whether it's correct**. Asked for the largest planet, GPT-2 small rated " Earth" at 0.0282, slightly above " Jupiter" at 0.0270. The number also moves with sampling settings, and the GPT-4 report found the model much less calibrated after post-training.

Instead, **use logprobs internally**: log them, flag low-probability answers for review, and check on a labelled set that the flag actually catches errors.

## [Write beam search that returns each sentence with its probability. How does it differ from greedy decoding?](https://llmdojo.dev/interview/q/write-beam-search-that-returns-each-sentence-with-its-probability-how-does)

Level: Mid-level

Keep the **k best partial sentences**. Each step, extend every one with its likely next tokens, score by the **sum of log-probabilities**, and keep the top k. A sentence's probability is the exponent of that sum.

**Diagram: Beam search with width 2**

1. Step 1: score the next token. With beam width 2, keep the two best, “cat” and “dog”, and drop the rest.
2. Step 2: extend each kept sentence and multiply the probabilities. In code, add log-probabilities instead, so long products don't underflow.
3. Keep the best two again. “dog barked” wins at 0.36, although “dog” was only the second-best first word.

Illustrative probabilities.

Greedy is beam width 1, so it misses sequences whose first token wasn't the best. Longer sentences collect more negative terms, so **normalise by length** when comparing finished beams.

## [Build a bigram next-word predictor that samples the next word in proportion to how often it followed.](https://llmdojo.dev/interview/q/build-a-bigram-next-word-predictor-that-samples-the-next-word-in)

Level: Junior

Count each word's successors, then **sample in proportion to the counts**.

```python

from collections import Counter, defaultdict

def train(words):
    counts = defaultdict(Counter)
    for a, b in zip(words, words[1:]):
        counts[a][b] += 1
    return counts

def next_word(counts, w):
    nxt = counts.get(w)
    if not nxt:
        return None              # unseen word
    ws, cs = zip(*nxt.items())
    return random.choices(ws, weights=cs)[0]
```

Interviewers check the unseen-word case and whether you can sample without random.choices: a **cumulative sum** plus binary search.

## [Cover the fundamentals: self- versus cross-attention, autoregressive decoding and sampling, where RL fits, and how you evaluate the result.](https://llmdojo.dev/interview/q/cover-the-fundamentals-self-versus-cross-attention-autoregressive-decoding-and-sampling-where)

Level: Mid-level

**Self-attention** draws queries, keys and values from one sequence; **cross-attention** takes queries from the decoder and keys and values from another source, like an encoder's output.

**Diagram: How a model writes, one token at a time**

1. The model reads the whole prompt: every position goes through all the transformer blocks.
2. To predict what comes next, only the vector at the last position is used.
3. The LM head turns that vector into one score per vocabulary token: a logit. Higher means more likely next.
4. Softmax turns the scores into probabilities. Temperature divides the scores first: below 1 sharpens the distribution, above 1 flattens it.
5. One token is chosen: greedy takes the most likely; sampling draws at random, weighted by probability, optionally only among the top-k or top-p candidates.
6. The chosen token is appended and the whole loop runs again. One loop per generated token, until a stop token or a length limit.

The standard autoregressive decoding loop of a GPT-style model. The vocabulary size is GPT-2's 50,257 tokens.

Decoding is a loop: score the next token, pick one greedily or by temperature and top-p, append, repeat, with a KV cache. RL then tunes those sampled outputs against a reward, held near a reference model. Evaluate with perplexity, task benchmarks and **human or judge preferences**.

## [Given logits from a toy language model, implement greedy, top-k and top-p decoding in NumPy.](https://llmdojo.dev/interview/q/given-logits-from-a-toy-language-model-implement-greedy-top-k-and)

Level: Junior

Greedy takes the argmax. Both samplers sort the probabilities, **keep a prefix**, renormalise it and sample.

```python

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def pick(logits, rng, mode="greedy", k=5, p=0.9,
         temp=1.0):
    if mode == "greedy":
        return int(np.argmax(logits))
    probs = softmax(logits / temp)
    order = np.argsort(probs)[::-1]   # high to low
    if mode == "top_k":
        keep = order[:k]
    else:                             # top_p
        cum = np.cumsum(probs[order])
        keep = order[:np.searchsorted(cum, p) + 1]
    q = probs[keep] / probs[keep].sum()
    return int(rng.choice(keep, p=q))
```

Interviewers check the **max subtracted** in softmax, that top-p keeps the token that crosses p, and the renormalisation before sampling.

## Related
- [All interview questions](https://llmdojo.dev/interview)