Byte pair encoding (BPE)
Also called byte pair encoding, BPE
A way to learn a subword vocabulary: start from single symbols and repeatedly merge the most frequent adjacent pair. Encoding replays the learned merges in order.
Interview questions about Byte pair encoding (BPE)
- How does byte pair encoding learn a vocabulary, and how does it then encode a word it never saw in training?
Training repeatedly merges the most frequent adjacent pair of symbols into a new one, giving an ordered merge list.
- Compare the main NLP tokenization approaches. Then, how would you design a recommender that uses an LLM?
Word vocabularies are huge and fail on unseen words; characters never fail but make sequences long.
- How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?
Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown.