You will be able to
- Log everything an agent does and link it into traces showing each step's inputs, outputs, time and cost.
- Read a failing trace to find the first wrong step.
- Name the agent failure modes: planning, goal, reflection, tool and efficiency failures.
- Compute tool-call metrics (valid calls, invalid tools, invalid parameters, wrong values) from logs, and act on the tools that fail most.
- Judge success by comparing the environment's end state with a goal state, not by the agent's claim.
- Measure consistency with pass^k alongside pass@k, and choose between them by how the agent is used.
- Build a sandboxed test harness that reruns on every change.
Topics
Your review
Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.
Sign in freeSources
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 6 "RAG and Agents": "Agent Failure Modes and Evaluation" ("Planning failures", "Tool failures", "Efficiency"), "Tool selection". Ch. 10 "AI Engineering Architecture and User Feedback": "Monitoring and Observability" > "Logs and traces".
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · Shunyu Yao et al., 2024
Shunyu Yao, Noah Shinn, Pedram Razavi & Karthik Narasimhan. arXiv:2406.12045. Used: Abstract; definition of pass^k (fetched 2026-10-23). Open source
Building effective agents · Erik Schluntz & Barry Zhang (Anthropic), 2024
Erik Schluntz & Barry Zhang (Anthropic). Used: "Agents" (fetched 2026-10-23). Open source