Special token
Also called special tokens
A vocabulary entry with a control meaning rather than text, such as end-of-text or a chat-role marker. User text must never become one.
Interview questions about Special token
- A ticket summariser crashes because tiktoken raises ValueError on a ticket that contains the literal end-of-text marker. A teammate's fix is to pass allowed_special='all'. What's your review comment?
Block it: that change is the injection. Special tokens like carry control meaning, so with allowedspecial="all" a user who types the marker can end a turn…
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
Start with what the model actually sees: decode a batch back to text.