Junior interview questionAsked at Google · interview write-up
Build a bigram next-word predictor that samples the next word in proportion to how often it followed.
Model answer
Count each word's successors, then sample in proportion to the counts.
import random
from collections import Counter, defaultdict
def train(words):
counts = defaultdict(Counter)
for a, b in zip(words, words[1:]):
counts[a][b] += 1
return counts
def next_word(counts, w):
nxt = counts.get(w)
if not nxt:
return None # unseen word
ws, cs = zip(*nxt.items())
return random.choices(ws, weights=cs)[0]
Interviewers check the unseen-word case and whether you can sample without random.choices: a cumulative sum plus binary search.
Practise the rest
9 Sampling questions, 492 in all, each with a model answer. Free.
More from Google interview questions.
More Sampling interview questions
- What are logits, and how does a model go from them to a whole sentence of output?
- Explain temperature, top-k and top-p. Why would you usually reach for top-p rather than a fixed k?
- Greedy decoding on a small model gives fluent text that soon repeats itself word for word. Why does that happen?
- Your integration tests call a hosted model at temperature 0 and compare the output with an exact string. They fail now and then. What's going on, and what do you change?
- A PM wants to show users a 'confidence' percentage computed from the answer's token probabilities. What's your position, and what would you build instead?
- Write beam search that returns each sentence with its probability. How does it differ from greedy decoding?