Senior interview question
You're deciding between a model API and self-hosting. How do you find the break-even, and what does the simple calculation leave out?
Model answer
Break-even requests per day = machine cost per day ÷ API cost per request, if one machine can serve that many. An illustrative $48-a-day machine against $0.002304 per request crosses at about 20,800 a day.
Left out: idle time (at 1,000 requests a day that machine costs $0.048 per request, 0.23% busy); quality, since our throughput is GPT-2 on a laptop, not a frontier model; engineering, which the book says can cost more than the API; and tokenizers, so compare per request. Re-run it as traffic grows.
Practise the rest
9 Cost per request questions, 492 in all, each with a model answer. Free.
More Cost per request interview questions
- How is an API request priced, and why is the reply often most of the bill?
- How do you turn a machine's hourly price into a cost per token, and how much can batching change it?
- (Illustrative numbers.) Model A costs $0.004 per request and is correct 50% of the time; model B costs $0.006 and is correct 95% of the time. Which is cheaper, and what should you track?
- How do you cut token costs at scale?
- When is a small open model good enough, rather than a frontier model?
- When do you send inference to a fast model and when to a cheap one? Walk me through the cost and latency trade-offs.