Topic 4

When the template is wrong

A wrong template rarely crashes anything. AI Engineering (Ch. 5, "System PromptSystem promptThe developer's instructions, sent as the first message with the system role. The model gets it in the same sequence as the user's text; it carries more weight only through position and training.Where this is taught and User Prompt") warns that template errors "cause silent failures—the model will do something reasonable even if the template is wrong", and that even "an extra new line" can significantly change behaviour.

The reason: a fine-tuned model has seen its markers in exactly one arrangement. Anything else is a sequence it was never trained on, even when it reads the same to you.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>↵
↵
↵
Translate the text below into French<|eot_id|><|start_header_id|>user<|end_header_id|>↵
↵
How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>↵
↵

What the lab shows, for the two-message chat:

• Extra newline: same 25 tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught, but token #4 is a different ID.
• No assistant header: 21 tokens ending on the user's turn. Hugging Face's guide says the model may then continue the user's message instead of replying.
• Begin-of-text twice: 26 tokens. The guide warns that adding special tokensSpecial tokenA vocabulary entry with a control meaning rather than text, such as end-of-text or a chat-role marker. User text must never become one.Where this is taught again on top of a template is "often incorrect or duplicated, hurting model performance".
• Another model's template: Zephyr's markers mean nothing to Llama 3.

The fix is mechanical: use the template that ships with the model (in Hugging Face, apply_chat_template), and test the token IDsToken IDThe integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.Where this is taught, not just the printed string.