Mid-level interview question
How should you judge whether an agent's task succeeded?
Model answer
By the end state of the environment, not the agent's claim. τ-bench compares the database after the conversation with an annotated goal state.
Run in a sandbox reset before each trial, and include what must not change, so side effects count as failures.
Practise the rest
6 Agent evaluation questions, 492 in all, each with a model answer. Free.
More Agent evaluation interview questions
- What is a trace, and what should it capture?
- How do you debug an agent run that claims success but did the wrong thing?
- What failure modes are specific to agents?
- Your agent succeeds 87.5% of the time on a task. Is it ready to run unattended?
- Your logs show most bad tool calls go to one tool. What do you do?