Agents interview questions

An LLM in a loop with tools: it picks the next step, calls a tool such as search or code execution, reads the result, and repeats until it judges the task done.

Unlike a fixed pipeline, the model controls the flow, so it handles open-ended tasks, and can fail in open-ended ways.

Neither by default. A fixed workflow, code that calls the model in set steps, is cheaper, faster and easier to test; use it when you know the steps. An agent earns its cost when the steps can't be known in advance. Start with the simplest thing that works, and add autonomy only where evals show it helps.

Profile one run: count the model calls, tokens per call and tool latency. Then run independent tool calls in parallel, use a small fast model for routing and simple steps, cache tool results and the static prompt prefix, trim the growing history, cap the steps, and turn predictable chains into plain code. Stream progress, so users see it working.

Checks outside the model on what goes in and comes out: input filters for injection or off-topic requests, validation of tool calls before they run (allowed tools, arguments, permissions), and output checks for policy, private data and format. The tool boundary matters most, because that's where an agent's mistakes become real actions.

Compounding errors: every step can fail, so long chains fail often; keep them short and check intermediate results. Loops and cost: cap steps, tokens and spend per task, and catch repeated identical calls. Latency grows with each call. Context fills with tool output, so summarise it. Unsafe actions: least-privilege tools, and a confirmation for anything irreversible. Trace every run, so failures can be replayed and turned into tests.

The harness owns everything the model doesn't. It runs the tool-call loop, executes tools in a sandbox with no network by default, scoped credentials and time limits, enforces step and token budgets, and gets human approval for irreversible actions. Every step is logged as a replayable trace.

Evaluation reuses that harness: a versioned task suite with deterministic graders where possible, such as tests passing or final state matching, rubric graders otherwise, and several runs per task because agents are stochastic. Report success, cost, steps and safety violations, and turn production failures into new tasks.

Let deterministic tools do what they do reliably: linters, type checkers, tests and static analysis run first, and their results go in as facts. The LLM reads the diff with retrieved context, such as callers, related files and the ticket, and comments on intent, logic and design that tools can't see.

Review per file or hunk, then one pass dedupes and ranks findings. Precision beats recall, since noisy bots get ignored: each comment must name a line and a concrete failure. Evaluate on past PRs against comments humans accepted, then track acceptance in production.

It's a loop: observe, decide, act, check. Perception pairs a screenshot with the accessibility tree when one exists; the tree gives element names and ids, which are far more reliable to target than raw coordinates. Without one, mark candidate elements on the image and let the model pick a mark.

Actions are typed tools: click, type, scroll, key press. After each, a fresh screenshot verifies the effect. Run in a sandboxed VM, treat on-screen text as data, not instructions, and ask the user before irreversible steps. Measure task success on benchmarks like OSWorld.

A pipeline with an agent in the middle. Triage first: take only tickets with a clear reproduction and a scoped component. The agent gathers context with tools: code search, file reads, git history, and tests run in a sandbox without production credentials. It writes a failing test, edits code and loops until tests pass, under step and cost limits.

The output is a pull request with a summary and links, always reviewed by a human.

Use an agent only where judgement is needed. Classify the ticket into service and issue type, which picks the playbook and tools. Then an investigation loop queries logs, metrics, traces, recent deploys and runbooks through read-only tools, with time windows anchored to the ticket and budgets on steps and tokens.

It returns a diagnosis with cited evidence, a confidence and next steps, which an engineer accepts or corrects. Evaluate on resolved past tickets: root-cause accuracy, whether cited evidence supports the claim, time to diagnosis, and how often engineers accept it.

An LLM planner turns the goal into structured intent: category, constraints, budget, occasion. It asks a clarifying question only when the answer would change the results, otherwise it searches with sensible defaults. Each step calls the existing search and filter APIs, inspects results and refines.

Deterministic logic wins wherever correctness is checkable: price, stock, delivery dates, compatibility, policy and safety rules. The LLM picks queries and explains choices; it never invents product facts. Cap the steps for latency, stream partial results, and measure task success and conversion against classic search.

Split the work by what can be checked. A classifier labels the document type, and the type selects a rule set. Checkable rules run as code: required clauses present, dates valid, amounts consistent. The model handles judgment rules, and every finding cites the rule id, the exact span and a proposed edit. Suggestions arrive as a reviewable diff; humans accept, reject or override, and their decisions are logged as eval data.

Don't put every rule into one prompt. First map each rule to the clauses it touches. Then, per rule, a checker decides compliant or not and cites the span, and a reviser edits only failing clauses. Fixes can conflict, so re-check every rule against the revised contract and loop until all pass or a budget runs out, escalating conflicts to a lawyer. Output a tracked-changes diff and per-rule report.

Build environments that mirror the work: a sandbox with a document store, search and editing tools, and tasks spanning dozens of steps over long files. Warm-start with supervised fine-tuning on expert trajectories, then RL. Prefer verifiable rewards, like the right clauses found or the final document passing checks; use a calibrated judge for the rest, a small step penalty, and a curriculum from short to long tasks. Read transcripts for reward hacking.

Working memory is the context window: task, recent turns, tool results. Session memory keeps the conversation plus a rolling summary. Long-term memory is episodic (past events), semantic (facts and preferences) and procedural (instructions and skills).

Writes: after a turn, an extraction step decides what's worth keeping, timestamps it and updates conflicting facts. Reads: retrieve by relevance, recency and importance, then fit the results into the prompt's budget.