# Tokenizer

> Tokenizer: The component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.

Canonical: https://llmdojo.dev/glossary/tokenizer

Also called tokenizers.

The component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.

Learn it in the lesson: [Models do not read text](https://llmdojo.dev/learn/model-foundations/tokenization/models-do-not-read-text) (Model Foundations · Tokenization: From Text to Token IDs).

## Interview questions about Tokenizer

- [How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?](https://llmdojo.dev/interview/q/how-would-you-design-a-subword-tokenizer-for-a-new-llm-and)
- [Implement a byte-pair encoding tokenizer with train, encode and decode, where training keeps merging the most frequent adjacent pair until counts fall below a threshold.](https://llmdojo.dev/interview/q/implement-a-byte-pair-encoding-tokenizer-with-train-encode-and-decode-where)
- [Before coding a BERT keyphrase span-extraction task, what would you clarify about the interface, tokenizer offsets, valid spans and the outputs?](https://llmdojo.dev/interview/q/before-coding-a-bert-keyphrase-span-extraction-task-what-would-you-clarify)
- [What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?](https://llmdojo.dev/interview/q/what-does-perplexity-measure-how-do-tokenizers-distort-it-and-where-does)
- [When you send a prompt to a language model, what does the model actually receive?](https://llmdojo.dev/interview/q/when-you-send-a-prompt-to-a-language-model-what-does-the)
- [What is a token, and why do most models use subword tokens rather than whole words or single characters?](https://llmdojo.dev/interview/q/what-is-a-token-and-why-do-most-models-use-subword-tokens)
- [A colleague estimates next month's API bill by counting the words in a batch of English and Hindi support tickets and multiplying by the price per token. What's wrong with the estimate, and how would you do it instead?](https://llmdojo.dev/interview/q/a-colleague-estimates-next-month-s-api-bill-by-counting-the-words)
- [You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?](https://llmdojo.dev/interview/q/you-re-moving-a-product-from-a-model-that-uses-cl100k-base)

## Related
- [Token](https://llmdojo.dev/glossary/token)
- [Token ID](https://llmdojo.dev/glossary/token-id)
- [Vocabulary](https://llmdojo.dev/glossary/vocabulary)
- [Byte pair encoding (BPE)](https://llmdojo.dev/glossary/bpe)
- [Subword](https://llmdojo.dev/glossary/subword)
- [Special token](https://llmdojo.dev/glossary/special-token)
- [Glossary of every term](https://llmdojo.dev/glossary)