Topic 6
Defences in layers
AI Engineering (Ch. 5, "Defenses Against Prompt Attacks") sorts defences into three levels. OWASP LLM01:2025 lists overlapping measures. Use several, because each covers a different step.
Scroll sideways to see the whole diagram
A model can be trained to rank instructions by where they come from, so a tool output loses to the system prompt.
Text version
- A model can be trained to rank instructions by where they come from, so a tool output loses to the system prompt.
- The prompt can name what is off-limits and mark untrusted text as data. Both rely on the model complying.
- System-level defences don't depend on the model: a tool it doesn't have can't be misused, and a send that needs approval can be declined.
- Layers reduce risk; none removes it. Plan for an attack that gets through.
Levels from AI Engineering, Ch. 5, "Defenses Against Prompt Attacks"; measures from OWASP LLM01:2025.
Model level. Wallace et al.'s instruction hierarchy (OpenAI, 2024), as the book describes it, ranks four sources: system prompt, user prompt, model outputs, tool outputs. When instructions conflict, the higher one wins, which "can neutralize many indirect prompt injection attacks". The book reports their fine-tuning increased robustness "by up to 63%" with minimal loss of capability.
Prompt level. Say what the model must not do, and separate untrusted text. OWASP: "Separate and clearly denote untrusted content to limit its influence." The book adds that "there's no guarantee that a model will follow the instructions given."
System level, in code the model can't talk its way past:
- Least privilege. OWASP: restrict the model's access "to the minimum necessary for its intended operations."
- Human approval. The book: don't let "potentially impactful commands" run "without explicit human approvals"; OWASP's measure 5 says the same.
- Input and output checks. The book: "inputs that appear harmless can produce harmful outputs", so guard both sides. OWASP adds validating output formats with deterministic code.
- Isolation. Run generated code in a separate virtual machine.
Quick check