Token ID
Also called token IDs
The integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.
Interview questions about Token ID
- You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?
Anything that stores tokens, because tokenizer mismatches fail silently: decoding IDs with the wrong tokenizer returns plausible garbage, not an error.
- After a library upgrade, your self-hosted Llama 3 gives worse answers and no errors. How do you debug it?
Suspect the chat template: its errors "cause silent failures". Diff the token IDs of one fixed prompt before and after; the printed string can't catch it.
- Turn an LLM into a binary classifier using its log-probs and log-sum-exp.
Don't parse generated text; read the next-token distribution after the prompt.