Mid-level interview question
What failure modes are specific to agents?
Model answer
- Planning: invalid tool, invalid parameters, wrong values.
- Goal: misses the task or breaks a constraint, including time.
- Reflection: claims it's done when it isn't.
- Tools: right tool, wrong output, or a missing tool.
- Efficiency: right result, too many steps or too costly.
Scroll sideways to see the whole diagram
AI Engineering: the most common planning failure is in tool use: a tool that doesn't exist, a real tool with the wrong parameters, or the right parameters with the wrong values.
Text version
- AI Engineering: the most common planning failure is in tool use: a tool that doesn't exist, a real tool with the wrong parameters, or the right parameters with the wrong values.
- Goal failures: the plan doesn't achieve the task, or does it while breaking a constraint. Its example: a trip to Ho Chi Minh City when you asked for Hanoi, or one way over budget. Time is a constraint too.
- Reflection failures: the agent believes it's done when it isn't. The book's example: asked to put 50 people in 30 hotel rooms, it places 40 and insists the task is complete.
- Tool failures: the tool itself returns something wrong, or the agent lacks the tool the task needs.
- And efficiency: steps, cost and time per task, compared with a baseline, which can be another agent or a person.
Practise the rest
6 Agent evaluation questions, 492 in all, each with a model answer. Free.
More Agent evaluation interview questions
- What is a trace, and what should it capture?
- How do you debug an agent run that claims success but did the wrong thing?
- How should you judge whether an agent's task succeeded?
- Your agent succeeds 87.5% of the time on a task. Is it ready to run unattended?
- Your logs show most bad tool calls go to one tool. What do you do?