Topic 5
Which instruction wins
A prompt mixes text from parties you trust differently: the developer, the user, and content the model reads, such as retrieved documents, web pages and tool results. When their instructions disagree, which wins?
OpenAI's paper "The Instruction Hierarchy" (Wallace et al., 2024, arXiv:2404.13208, fetched 2026-10-21) argues that many attacks work because models treat these all alike. It proposes an order: "system messages take precedence over user messages, and user messages take precedence over third-party content". AI Engineering (Ch. 5, "Model-level defense") lists the four levels as system prompt, user prompt, model outputs, tool outputs.
Predict first
Scroll sideways to see the whole diagram
The developer's system message sets the job and the tools. It has the highest priority.
Text version
- The developer's system message sets the job and the tools. It has the highest priority.
- The user asks for something the system allows: read the latest email. Aligned, so it's followed.
- The model calls the read tool. Its own outputs rank third.
- The email comes back as tool output, and it contains an instruction. In the paper's example, a model without this training obeys and forwards every email.
- Under the hierarchy, tool output ranks last and this instruction conflicts with what the user asked, so it should be ignored.
Example and priority order: Wallace et al. 2024, "The Instruction Hierarchy" (arXiv:2404.13208), as quoted in AI Engineering, Ch. 5 "Defensive Prompt Engineering": "Indirect prompt injection" and "Model-level defense".
Two refinements from the paper (§3.1). Lower-level instructions that agree with higher ones (aligned) should still be followed: "use a formal tone" from the user is fine. Only misaligned ones are ignored. And this is learned behaviour: training GPT-3.5 this way improved defence against system-prompt extraction by 63% (§1), which is better, not perfect.
The template has to cooperate. Meta's code encodes message content as plain text, so a user who types <|eot_id|> cannot close their own turn (try the "User types a marker" preset in the assembler). Module 01 covers why: special-token injection.
Quick check