Cost per Request and per Useful Answer

You will be able to

• Compute an API request's cost from its input and output token counts and their separate prices, and explain why output usually dominates.
• Turn a machine's hourly price and throughput into a cost per million tokens, and show from measured throughput how batching lowers it.
• Explain why idle time raises the cost per request of a self-hosted model, and estimate it from utilisation.
• Compute cost per useful answer from the share of requests that meet the SLO and the share that are correct, including the effect of retries.
• Explain why prices per token can't be compared across tokenizers, and compare cost per request instead.
• Estimate the API versus self-host break-even with stated assumptions, and name the costs it leaves out.

Topics

1. 1Two prices per request
2. 2Self-hosting: paying by the hour
3. 3Batching cuts the cost per token
4. 4Idle time costs money
5. 5Cost per useful answer
6. 6Compare per request, not per token
7. 7API or self-host: the break-even