Vocabulary
The fixed list of every token a tokenizer knows. Its size is a design choice made before training.
Interview questions about Vocabulary
- How does byte pair encoding learn a vocabulary, and how does it then encode a word it never saw in training?
Training repeatedly merges the most frequent adjacent pair of symbols into a new one, giving an ordered merge list.
- An input embedding table is too large for one device. Compare sharding it by vocabulary with sharding it by hidden dimension.
By vocabulary, each device owns a range of rows, looks up the tokens it owns, writes zeros for the rest, and an all-reduce sum assembles full vectors.
- Your LM head is too big for one device. How do you shard it by vocabulary or by hidden width, combine the logits, and compute an exact top-k across shards?
Vocabulary sharding: each device holds a slice of rows and computes logits for its slice only.
- When you send a prompt to a language model, what does the model actually receive?
A list of integer IDs, not the string. A tokenizer swaps each piece of text for an ID from a fixed vocabulary; the model reads IDs and writes them one at a…
- What is a token, and why do most models use subword tokens rather than whole words or single characters?
A token is a piece of text from the tokenizer's fixed vocabulary, stored as an integer ID.
- Compare the main NLP tokenization approaches. Then, how would you design a recommender that uses an LLM?
Word vocabularies are huge and fail on unseen words; characters never fail but make sequences long.
- How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?
Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown.
- Before coding a BERT keyphrase span-extraction task, what would you clarify about the interface, tokenizer offsets, valid spans and the outputs?
Interface: one keyphrase per text or several, overlapping or not, labelled as offsets or strings? Tokenizer: which vocabulary, cased or not.