Subword
Also called subwords
A token that can be a whole word or a fragment of one. Common words get one token; rare words are built from several.
Interview questions about Subword
- What is a token, and why do most models use subword tokens rather than whole words or single characters?
A token is a piece of text from the tokenizer's fixed vocabulary, stored as an integer ID.
- How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?
Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown.
- What is subword tokenization, what does SentencePiece add, and what are the main transformer variants?
Subword tokenizers keep common words whole and split rare ones into frequent pieces: no unknown words, short sequences.
- Compare the main NLP tokenization approaches. Then, how would you design a recommender that uses an LLM?
Word vocabularies are huge and fail on unseen words; characters never fail but make sequences long.
- Before coding a BERT keyphrase span-extraction task, what would you clarify about the interface, tokenizer offsets, valid spans and the outputs?
Interface: one keyphrase per text or several, overlapping or not, labelled as offsets or strings? Tokenizer: which vocabulary, cased or not.
- Compare tokenization choices and dense versus sparse retrieval for search and RAG, then design and evaluate a hybrid retriever.
Sparse retrieval like BM25 matches tokens, so its tokenization matters: words with stemming suit English prose, while code, part numbers and other languages…