Senior interview questionAsked at Anthropic · interview write-up
Design a backend where clients submit LLM prompt jobs and poll or receive callbacks, with queueing, model routing, prompt versioning, idempotency and cost control.
Model answer
Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry. Jobs queue per tenant and priority, with quotas against starvation.
Workers resolve the prompt version pinned in the request, route to a model by task and budget, and store results; clients poll or get a signed webhook. Control cost with token estimates at admission, budget caps, batch APIs for non-urgent work, prompt caching and per-tenant metering. Failures back off, then dead-letter.
Practise the rest
9 Cost per request questions, 492 in all, each with a model answer. Free.
More from Anthropic interview questions.
More Cost per request interview questions
- How is an API request priced, and why is the reply often most of the bill?
- How do you turn a machine's hourly price into a cost per token, and how much can batching change it?
- (Illustrative numbers.) Model A costs $0.004 per request and is correct 50% of the time; model B costs $0.006 and is correct 95% of the time. Which is cheaper, and what should you track?
- You're deciding between a model API and self-hosting. How do you find the break-even, and what does the simple calculation leave out?
- How do you cut token costs at scale?
- When is a small open model good enough, rather than a frontier model?