Top-k
Sampling only among the k most likely tokens, after renormalising their probabilities.
Interview questions about Top-k
- Explain temperature, top-k and top-p. Why would you usually reach for top-p rather than a fixed k?
Temperature divides every logit by T before softmax: low T sharpens, high T flattens, and T near 0 approaches greedy.
- Given logits from a toy language model, implement greedy, top-k and top-p decoding in NumPy.
Greedy takes the argmax. Both samplers sort the probabilities, keep a prefix, renormalise it and sample.
- Your LM head is too big for one device. How do you shard it by vocabulary or by hidden width, combine the logits, and compute an exact top-k across shards?
Vocabulary sharding: each device holds a slice of rows and computes logits for its slice only.
- Design a real-time system that shows the top-K points of interest in a user's map viewport as they pan and zoom, within a 150 ms p95 budget.
Split it into spatial retrieval and ranking, both cheap. Index POIs in a spatial grid such as geohash or S2 cells, and precompute the top candidates per cell…
- Your assistant's semantic search over user content is slow and expensive. How do you optimise it: embedding choice, ANN indexing and sharding?
Embeddings: shortlist models on public retrieval benchmarks, then pick on your own queries; smaller dimensions or int8 vectors cut memory and latency.
- Your integration tests call a hosted model at temperature 0 and compare the output with an exact string. They fail now and then. What's going on, and what do you change?
The test assumes a determinism a hosted API doesn't promise. Temperature 0 is greedy and uses no random numbers, and our sampling script gave byte-identical…