Senior interview question

Your agent succeeds 87.5% of the time on a task. Is it ready to run unattended?

Q: Your agent succeeds 87.5% of the time on a task. Is it ready to run unattended?

Measure pass^k: the chance that all k trials succeed, unlike pass@k (at least one). On the course's illustrative trials, pass@8 is 0.833 but pass^8 is 0.167. τ-bench found even gpt-4o under 50% on its tasks, with pass^8 below 25% in retail. Unattended agents need consistency, not just a success rate.

Model answer

Measure pass^k: the chance that all k trials succeed, unlike pass@k (at least one). On the course's illustrative trials, pass@8 is 0.833 but pass^8 is 0.167.

τ-bench found even gpt-4o under 50% on its tasks, with pass^8 below 25% in retail. Unattended agents need consistency, not just a success rate.

Practise the rest

6 Agent evaluation questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Agent evaluation

What is a trace, and what should it capture?
How do you debug an agent run that claims success but did the wrong thing?
What failure modes are specific to agents?
How should you judge whether an agent's task succeeded?
Your logs show most bad tool calls go to one tool. What do you do?

Model answer

More Agent evaluation interview questions

Model answer

More Agent evaluation interview questions