Cost per request
What one request costs end to end: input tokens × input price + output tokens × output price for an API, or machine time × hourly price when self-hosting. Comparable across models with different tokenizers, unlike a price per token.
Interview questions about Cost per request
- You're deciding between a model API and self-hosting. How do you find the break-even, and what does the simple calculation leave out?
Break-even requests per day = machine cost per day ÷ API cost per request, if one machine can serve that many.
- When do you send inference to a fast model and when to a cheap one? Walk me through the cost and latency trade-offs.
Route by difficulty and latency budget: easy, high-volume requests to the small model, hard ones to the large.
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
Measure it myself: their number is their workload on their day.