Topic 7

Every token costs, and wrap-up

Every tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught of a prompt is paid for on every call: in input price, and in prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught time before the first token. AI Engineering (Ch. 5, "Provide examples") advises choosing example formats that use fewer tokens, if they perform equally well.

The template costs tokens too: 15 of the 25 in the two-message chat on the chat-templates page. You can't remove those, but you can make the rest cheaper to repeat. A prompt cachePrompt cachingKeeping the KV cache of a prompt's shared opening, such as a system prompt or long document, so later requests that start with the identical prefix skip prefilling it.Where this is taught reuses only an identical prefix, so put the system promptSystem promptThe developer's instructions, sent as the first message with the system role. The model gets it in the same sequence as the user's text; it carries more weight only through position and training.Where this is taught and examples first and the per-request part last: the prefix must match exactly.

What to keep

• A prompt is task description, examples and the task, all in one sequence.
• Roles are markers in that sequence, built by the model's own chat templateChat templateThe model-specific rule that turns a list of role-tagged messages into the one token sequence the model was fine-tuned on, with special markers around each turn. A wrong template fails silently.Where this is taught. Use it exactly, and check token IDsToken IDThe integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.Where this is taught, not just the printed string.
• System beats user beats tool output only because models are trained that way; misaligned lower-level instructions should be ignored.
• Be specific: scale, format, markers, persona.
• Count tokens, and order the prompt static-first.