Traces and evaluating agents interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 traces and evaluating agents questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Traces and Evaluating Agents teaches the ideas behind them.Updated
Model answer
The linked record of one request's path through the system, every step, so a failure can be pinned to the exact step.
Scroll sideways to see the whole diagram
A trace is one request's full path through the system. The top line is the whole task: duration, cost and outcome (figures illustrative).
Text version
- A trace is one request's full path through the system. The top line is the whole task: duration, cost and outcome (figures illustrative).
- Inside it, each model call and each tool call is its own record, with its inputs and outputs, nested under what caused it.
- That's how the wrong charge ID becomes visible: it's right there in the arguments of one tool call.
- AI Engineering: a trace should show the whole process, how long each step takes and what it costs, so if a query fails you can pinpoint the step where it went wrong.
Log everything: model and settings, prompt version, inputs and outputs, every tool call with its arguments and result, timings and costs.
Model answer
Walk the trace from the start and find the first step whose inputs don't follow from earlier results. In the course's example, the agent refunded ch_1 when the duplicate was ch_2: a valid tool with a wrong value. Its claim of success is a separate reflection error.
Model answer
- Planning: invalid tool, invalid parameters, wrong values.
- Goal: misses the task or breaks a constraint, including time.
- Reflection: claims it's done when it isn't.
- Tools: right tool, wrong output, or a missing tool.
- Efficiency: right result, too many steps or too costly.
Scroll sideways to see the whole diagram
AI Engineering: the most common planning failure is in tool use: a tool that doesn't exist, a real tool with the wrong parameters, or the right parameters with the wrong values.
Text version
- AI Engineering: the most common planning failure is in tool use: a tool that doesn't exist, a real tool with the wrong parameters, or the right parameters with the wrong values.
- Goal failures: the plan doesn't achieve the task, or does it while breaking a constraint. Its example: a trip to Ho Chi Minh City when you asked for Hanoi, or one way over budget. Time is a constraint too.
- Reflection failures: the agent believes it's done when it isn't. The book's example: asked to put 50 people in 30 hotel rooms, it places 40 and insists the task is complete.
- Tool failures: the tool itself returns something wrong, or the agent lacks the tool the task needs.
- And efficiency: steps, cost and time per task, compared with a baseline, which can be another agent or a person.
Model answer
By the end state of the environment, not the agent's claim. τ-bench compares the database after the conversation with an annotated goal state.
Run in a sandbox reset before each trial, and include what must not change, so side effects count as failures.
Model answer
Measure pass^k: the chance that all k trials succeed, unlike pass@k (at least one). On the course's illustrative trials, pass@8 is 0.833 but pass^8 is 0.167.
τ-bench found even gpt-4o under 50% on its tasks, with pass^8 below 25% in retail. Unattended agents need consistency, not just a success rate.
Model answer
Improve its description and examples, fine-tune if needed, and if it's still misused, change the tool: simpler parameters or a narrower purpose.
Also look at the tool-use distribution and ablate: if removing a tool doesn't hurt, remove it. Then rerun the test harness on every change.