# Traces and evaluating agents interview questions > 6 traces and evaluating agents interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/agent-evaluation Last updated: 2026-10-23 6 questions: 1 junior, 3 mid-level and 2 senior. Taught in [Traces and Evaluating Agents](https://llmdojo.dev/learn/agents-harness-engineering/traces-and-evaluation). ## [What is a trace, and what should it capture?](https://llmdojo.dev/interview/q/what-is-a-trace-and-what-should-it-capture) Level: Junior The linked record of one request's path through the system, every step, so a failure can be pinned to the exact step. **Diagram: One request, as a trace** 1. A trace is one request's full path through the system. The top line is the whole task: duration, cost and outcome (figures illustrative). 2. Inside it, each model call and each tool call is its own record, with its inputs and outputs, nested under what caused it. 3. That's how the wrong charge ID becomes visible: it's right there in the arguments of one tool call. 4. AI Engineering: a trace should show the whole process, how long each step takes and what it costs, so if a query fails you can pinpoint the step where it went wrong. Log everything: model and settings, prompt version, inputs and outputs, every tool call with its **arguments** and result, timings and costs. ## [How do you debug an agent run that claims success but did the wrong thing?](https://llmdojo.dev/interview/q/how-do-you-debug-an-agent-run-that-claims-success-but-did) Level: Mid-level Walk the trace from the start and find the **first** step whose inputs don't follow from earlier results. In the course's example, the agent refunded `ch_1` when the duplicate was `ch_2`: a valid tool with a **wrong value**. Its claim of success is a separate reflection error. ## [What failure modes are specific to agents?](https://llmdojo.dev/interview/q/what-failure-modes-are-specific-to-agents) Level: Mid-level - **Planning**: invalid tool, invalid parameters, wrong values. - **Goal**: misses the task or breaks a constraint, including time. - **Reflection**: claims it's done when it isn't. - **Tools**: right tool, wrong output, or a missing tool. - **Efficiency**: right result, too many steps or too costly. **Diagram: How agents fail** 1. AI Engineering: the most common planning failure is in tool use: a tool that doesn't exist, a real tool with the wrong parameters, or the right parameters with the wrong values. 2. Goal failures: the plan doesn't achieve the task, or does it while breaking a constraint. Its example: a trip to Ho Chi Minh City when you asked for Hanoi, or one way over budget. Time is a constraint too. 3. Reflection failures: the agent believes it's done when it isn't. The book's example: asked to put 50 people in 30 hotel rooms, it places 40 and insists the task is complete. 4. Tool failures: the tool itself returns something wrong, or the agent lacks the tool the task needs. 5. And efficiency: steps, cost and time per task, compared with a baseline, which can be another agent or a person. ## [How should you judge whether an agent's task succeeded?](https://llmdojo.dev/interview/q/how-should-you-judge-whether-an-agent-s-task-succeeded) Level: Mid-level By the **end state of the environment**, not the agent's claim. τ-bench compares the database after the conversation with an annotated goal state. Run in a sandbox reset before each trial, and include what must **not** change, so side effects count as failures. ## [Your agent succeeds 87.5% of the time on a task. Is it ready to run unattended?](https://llmdojo.dev/interview/q/your-agent-succeeds-87-5-of-the-time-on-a-task-is) Level: Senior Measure **pass^k**: the chance that **all** k trials succeed, unlike pass@k (at least one). On the course's illustrative trials, pass@8 is 0.833 but pass^8 is 0.167. τ-bench found even gpt-4o under 50% on its tasks, with pass^8 below 25% in retail. Unattended agents need consistency, not just a success rate. ## [Your logs show most bad tool calls go to one tool. What do you do?](https://llmdojo.dev/interview/q/your-logs-show-most-bad-tool-calls-go-to-one-tool-what) Level: Senior Improve its description and examples, fine-tune if needed, and if it's still misused, **change the tool**: simpler parameters or a narrower purpose. Also look at the tool-use distribution and ablate: if removing a tool doesn't hurt, remove it. Then rerun the test harness on every change. ## Related - [All interview questions](https://llmdojo.dev/interview)