Retries, timeouts and idempotency interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 retries, timeouts and idempotency questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson When Steps Fail: Retries, Timeouts, Idempotency and Checkpoints teaches the ideas behind them.Updated
Model answer
Transient ones: timeouts and dropped connections, with backoff. Not permanent ones like invalid arguments: return those to the model so it can fix the call. Overload responses mean slow down, not retry harder.
Scroll sideways to see the whole diagram
Not every failure deserves the same response.
Text version
- Not every failure deserves the same response.
- Transient failures may succeed if tried again a little later: retry, with backoff.
- Overload errors mean the service is struggling. Google's SRE book: return a specific overload status so callers back off instead of retrying.
- Permanent errors will never succeed: malformed requests, invalid arguments. Retrying the identical call only adds load.
- In an agent, many permanent errors come from the model's arguments. Returning the error to the model lets it correct the call.
- AI Engineering's tool failures include a tool simply returning the wrong output: no error is raised. Catching those needs checks on results (lesson 5).
Model answer
Nothing certain: the request may have been lost, may still be running, or may have succeeded with the response lost. They look identical.
Scroll sideways to see the whole diagram
Every call needs a deadline, or one stuck tool stalls the whole agent.
Text version
- Every call needs a deadline, or one stuck tool stalls the whole agent.
- But when the deadline passes, three different things might have happened.
- The harness can't tell them apart. For a read, retrying is harmless. For a write like a refund, retrying blindly can do it twice: the next pages.
For reads, retry. For writes, retry only if the operation is idempotent, or check its effect first.
Model answer
Clients that failed together retry together. Backoff alone keeps them in step, so a recovering service gets all of them at once: in the course's simulation, 100 calls in one 100 ms window.
Random waits spread them out: peak 11.
Model answer
Attempts per layer raised to the number of layers: five layers of 3 is 243 attempts at the bottom.
Retry at one layer, cap retries per request, keep a process-wide retry budget, and count the model's own re-calls against it.
Model answer
Idempotency keys: the server stores the first result for a key and returns it to repeats, so a retried refund happens once.
The harness derives the key from task and step (or saves it in the checkpoint), so retries reuse it; a key the model invents each time protects nothing. Without API support, record completed writes and check before repeating.
Model answer
Checkpoint after every step: task, context or summary, results, completed side effects, keys in flight. On restart, load it and continue; the harness, not the model, decides which steps are done, so completed writes are skipped.
The same mechanism makes pauses for human approval cheap.