Topic 2
System, user and assistant messages
Chat APIs take a list of messages, each with a role:
- system: the developer's instructions, usually first. AI Engineering calls it the task description.
- user: what the user sends. It holds the task.
- assistant: the model's earlier replies, or an example reply you write yourself.
The book's example (Ch. 5, "System Prompt and User Prompt"): a chatbot that explains property disclosures. The system prompt makes it "an experienced real estate agent"; the user prompt carries the uploaded disclosure and the question.
Predict first
Scroll sideways to see the whole diagram
An API takes a list of messages, each with a role. This is the book's system and user prompt.
Text version
- An API takes a list of messages, each with a role. This is the book's system and user prompt.
- Before the model sees anything, a chat template turns the list into one sequence.
- Llama 3's template: begin-of-text, a header naming each role, a blank line (↵↵), the content, an end-of-turn marker.
- Each <|…|> marker is one special token. In a short chat, much of the sequence is the template's own.
- The sequence ends inside an assistant turn, so the most likely continuation is the assistant's reply.
Format: Meta's Llama 3 ChatFormat (github.com/meta-llama/llama3, llama/tokenizer.py, fetched 2026-10-21). No counts shown here; the next page counts them.
So roles are a convention the model learned, written into the sequence as markers. That has two consequences, each with its own page: the markers must be exactly the ones the model was trained on, and "the system prompt wins" is a trained behaviour, not a guarantee.
Quick check