Token
Also called tokens
A piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.
Interview questions about Token
- What is a token, and why do most models use subword tokens rather than whole words or single characters?
A token is a piece of text from the tokenizer's fixed vocabulary, stored as an integer ID.
- A colleague estimates next month's API bill by counting the words in a batch of English and Hindi support tickets and multiplying by the price per token. What's wrong with the estimate, and how would you do it instead?
Prices are per token of one specific tokenizer, and words-to-tokens isn't constant.
- Product wants to double the context you support, from 1,000 to 2,000 tokens. From the attention side, what gets more expensive, and what families of fixes would you weigh?
Attention compute grows quadratically. Each head scores an n × n grid, so doubling n roughly quadruples that work; the paper puts self-attention per layer at…
- A PM wants to show users a 'confidence' percentage computed from the answer's token probabilities. What's your position, and what would you build instead?
Against it: a token probability says how typical the text is after the prompt, not whether it's correct.
- What is a context window, and why can't GPT-2 small take a 1,025th token even with plenty of memory?
It's the most tokens a model can read at once, shared by prompt, history and output: 1,024 for GPT-2 small.
- Request A has a 960-token prompt and 10 output tokens. Request B has a 16-token prompt and 200 output tokens. Same model and machine: which finishes first, and what does that tell you about a chat app that resends the full history every turn?
A. On our GPT-2 small measurements, A is about 220 ms of prefill plus 10 steps of about 10 ms, roughly 0.3 s; B is about 15 ms plus 200 sequential steps of…
- Implement causal multi-head attention, add a KV cache for token-by-token decoding, then extend it to grouped-query attention.
Write it once as GQA: plain MHA is the case where key-value heads equal query heads, and multi-query is one key-value head.
- What is a hallucination, and why can a next-token predictor produce one so fluently?
It's fluent, confident output that is false or not grounded in any source.