Senior interview questionAsked at OpenAI · interview write-up
Design a harness for running a tool-using agent safely, and an evaluation platform on top of it.
Model answer
The harness owns everything the model doesn't. It runs the tool-call loop, executes tools in a sandbox with no network by default, scoped credentials and time limits, enforces step and token budgets, and gets human approval for irreversible actions. Every step is logged as a replayable trace.
Scroll sideways to see the whole diagram
The context holds the tool definitions and the user's question.
Text version
- The context holds the tool definitions and the user's question.
- The model emits a structured call: a tool name and JSON arguments. With Anthropic's API the reply stops with stop_reason "tool_use".
- The call is text the model generated. Your code checks it before anything runs.
- Your code executes the function. The model has no hands: it only asked.
- The output goes back into the context (a tool_result block) and the model runs again, until it answers instead of calling.
After AI Engineering, Ch. 6 "Function calling", and Anthropic, "Tool use with Claude" (fetched 2026-10-21). Illustrative call.
Evaluation reuses that harness: a versioned task suite with deterministic graders where possible, such as tests passing or final state matching, rubric graders otherwise, and several runs per task because agents are stochastic. Report success, cost, steps and safety violations, and turn production failures into new tasks.
Practise the rest
15 Agents questions, 492 in all, each with a model answer. Free.
More from OpenAI interview questions.
More Agents interview questions
- What actually is an AI agent?
- LLM workflow or agent: which is better?
- Your agent is very slow. What can you do?
- What are guardrails, and where do they sit in an agent?
- What are the bottlenecks of AI agents, and how do you keep them in check?
- Design a code-review agent that combines LLM orchestration with deterministic tools.