Senior interview questionAsked at Amazon · interview write-up
Design a computer-use agent end to end: how it perceives the screen from pixels and accessibility trees, and how it takes sequential actions.
Model answer
It's a loop: observe, decide, act, check. Perception pairs a screenshot with the accessibility tree when one exists; the tree gives element names and ids, which are far more reliable to target than raw coordinates. Without one, mark candidate elements on the image and let the model pick a mark.
Scroll sideways to see the whole diagram
The context holds the tool definitions and the user's question.
Text version
- The context holds the tool definitions and the user's question.
- The model emits a structured call: a tool name and JSON arguments. With Anthropic's API the reply stops with stop_reason "tool_use".
- The call is text the model generated. Your code checks it before anything runs.
- Your code executes the function. The model has no hands: it only asked.
- The output goes back into the context (a tool_result block) and the model runs again, until it answers instead of calling.
After AI Engineering, Ch. 6 "Function calling", and Anthropic, "Tool use with Claude" (fetched 2026-10-21). Illustrative call.
Actions are typed tools: click, type, scroll, key press. After each, a fresh screenshot verifies the effect. Run in a sandboxed VM, treat on-screen text as data, not instructions, and ask the user before irreversible steps. Measure task success on benchmarks like OSWorld.
Practise the rest
15 Agents questions, 492 in all, each with a model answer. Free.
More from Amazon interview questions.
More Agents interview questions
- What actually is an AI agent?
- LLM workflow or agent: which is better?
- Your agent is very slow. What can you do?
- What are guardrails, and where do they sit in an agent?
- What are the bottlenecks of AI agents, and how do you keep them in check?
- Design a harness for running a tool-using agent safely, and an evaluation platform on top of it.