Junior interview question
What is a trace, and what should it capture?
Model answer
The linked record of one request's path through the system, every step, so a failure can be pinned to the exact step.
Scroll sideways to see the whole diagram
A trace is one request's full path through the system. The top line is the whole task: duration, cost and outcome (figures illustrative).
Text version
- A trace is one request's full path through the system. The top line is the whole task: duration, cost and outcome (figures illustrative).
- Inside it, each model call and each tool call is its own record, with its inputs and outputs, nested under what caused it.
- That's how the wrong charge ID becomes visible: it's right there in the arguments of one tool call.
- AI Engineering: a trace should show the whole process, how long each step takes and what it costs, so if a query fails you can pinpoint the step where it went wrong.
Log everything: model and settings, prompt version, inputs and outputs, every tool call with its arguments and result, timings and costs.
Practise the rest
6 Agent evaluation questions, 492 in all, each with a model answer. Free.
More Agent evaluation interview questions
- How do you debug an agent run that claims success but did the wrong thing?
- What failure modes are specific to agents?
- How should you judge whether an agent's task succeeded?
- Your agent succeeds 87.5% of the time on a task. Is it ready to run unattended?
- Your logs show most bad tool calls go to one tool. What do you do?