Topic 5
Tool output is untrusted
Everything a tool returns goes into the context: search results, web pages, files, emails, tickets. Much of it was written by someone you don't control. Module 03 called this indirect prompt injection: instructions hidden in content the model reads, which it may follow as if they came from you.
Agents raise the stakes. A chatbot that follows injected text says something wrong. An agent that follows it does something: sends an email, changes a record, runs code.
Predict first
Scroll sideways to see the whole diagram
Anything a tool reads (a ticket, a web page, a document, an email) can contain text written by someone else, including instructions.
Text version
- Anything a tool reads (a ticket, a web page, a document, an email) can contain text written by someone else, including instructions.
- Once read, that text sits in the context alongside your instructions. This is the indirect prompt injection of Module 03.
- There's no reliable way to make the model ignore it, so assume it sometimes won't.
- The defence that holds is in the harness: limits on what any action can do, checked in code.
- The policy doesn't need to detect the attack. It only needs to forbid the outcome.
Treat every tool result as untrusted input, the same way Module 03 treated tool arguments:
- Mark it. Keep tool output clearly separated from instructions in the context, so the model has the best chance of treating it as data. It helps, but it's not a guarantee.
- Don't let it widen permissions. Nothing a tool returns should change which tools are available or what the policy allows.
- Check consequences, not content. The policy blocked the injected email by its recipient, without detecting the attack. Detection fails; outcome rules don't.
- Be wary of chains. An agent that reads a web page, then runs code, then sends a message, has given a stranger's text a route to every one of those tools.
Quick check