Mid-level interview questionAsked at Google · interview write-up
When do you send inference to a fast model and when to a cheap one? Walk me through the cost and latency trade-offs.
Model answer
Route by difficulty and latency budget: easy, high-volume requests to the small model, hard ones to the large. The router can be rules, a small classifier, or a cascade that tries the cheap model and escalates when a check fails, which adds a second call's latency on escalations.
Scroll sideways to see the whole diagram
A request has two token counts: the prompt you send and the reply the model writes.
Text version
- A request has two token counts: the prompt you send and the reply the model writes.
- Input tokens are billed at the input price: 2,000 tokens at $2 per million is $0.004.
- Output tokens have their own, higher price: 500 tokens at $10 per million is $0.005.
- Add the two. The reply is a fifth of the tokens but more than half the bill, so trimming replies often saves the most.
Claude Sonnet 5.5 list prices, $2 input / $10 output per million tokens: platform.claude.com/docs/en/about-claude/pricing, fetched 2026-10-21. Token counts illustrative.
Set thresholds on a labelled set by plotting quality against cost per request, and watch p95, where cascades hurt.
Practise the rest
9 Cost per request questions, 492 in all, each with a model answer. Free.
More from Google interview questions.
More Cost per request interview questions
- How is an API request priced, and why is the reply often most of the bill?
- How do you turn a machine's hourly price into a cost per token, and how much can batching change it?
- (Illustrative numbers.) Model A costs $0.004 per request and is correct 50% of the time; model B costs $0.006 and is correct 95% of the time. Which is cheaper, and what should you track?
- You're deciding between a model API and self-hosting. How do you find the break-even, and what does the simple calculation leave out?
- How do you cut token costs at scale?
- When is a small open model good enough, rather than a frontier model?