# Vocabulary

> Vocabulary: The fixed list of every token a tokenizer knows. Its size is a design choice made before training.

Canonical: https://llmdojo.dev/glossary/vocabulary

The fixed list of every token a tokenizer knows. Its size is a design choice made before training.

Learn it in the lesson: [Why subwords](https://llmdojo.dev/learn/model-foundations/tokenization/why-subwords) (Model Foundations · Tokenization: From Text to Token IDs).

## Interview questions about Vocabulary

- [How does byte pair encoding learn a vocabulary, and how does it then encode a word it never saw in training?](https://llmdojo.dev/interview/q/how-does-byte-pair-encoding-learn-a-vocabulary-and-how-does-it)
- [An input embedding table is too large for one device. Compare sharding it by vocabulary with sharding it by hidden dimension.](https://llmdojo.dev/interview/q/an-input-embedding-table-is-too-large-for-one-device-compare-sharding)
- [Your LM head is too big for one device. How do you shard it by vocabulary or by hidden width, combine the logits, and compute an exact top-k across shards?](https://llmdojo.dev/interview/q/your-lm-head-is-too-big-for-one-device-how-do-you)
- [When you send a prompt to a language model, what does the model actually receive?](https://llmdojo.dev/interview/q/when-you-send-a-prompt-to-a-language-model-what-does-the)
- [What is a token, and why do most models use subword tokens rather than whole words or single characters?](https://llmdojo.dev/interview/q/what-is-a-token-and-why-do-most-models-use-subword-tokens)
- [Compare the main NLP tokenization approaches. Then, how would you design a recommender that uses an LLM?](https://llmdojo.dev/interview/q/compare-the-main-nlp-tokenization-approaches-then-how-would-you-design-a)
- [How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?](https://llmdojo.dev/interview/q/how-would-you-design-a-subword-tokenizer-for-a-new-llm-and)
- [Before coding a BERT keyphrase span-extraction task, what would you clarify about the interface, tokenizer offsets, valid spans and the outputs?](https://llmdojo.dev/interview/q/before-coding-a-bert-keyphrase-span-extraction-task-what-would-you-clarify)

## Related
- [Token](https://llmdojo.dev/glossary/token)
- [Tokenizer](https://llmdojo.dev/glossary/tokenizer)
- [Token ID](https://llmdojo.dev/glossary/token-id)
- [Byte pair encoding (BPE)](https://llmdojo.dev/glossary/bpe)
- [Subword](https://llmdojo.dev/glossary/subword)
- [Special token](https://llmdojo.dev/glossary/special-token)
- [Glossary of every term](https://llmdojo.dev/glossary)