Junior interview question
What three budgets does an AI feature have?
Model answer
Cost, rate and latency. Each is a requirement enforced in code with a planned behaviour when it's hit. A provider's spend cap stops every feature on the account until it resets.
Scroll sideways to see the whole diagram
Cost: what each request, each user and the whole feature may spend. Module 02 showed how to compute it.
Text version
- Cost: what each request, each user and the whole feature may spend. Module 02 showed how to compute it.
- Rate: how many requests and tokens per minute, both the limits a provider imposes on you and the ones you impose on your users.
- Latency: how long a user waits, for the first token and for the whole answer.
- A budget that's only a number in a spreadsheet isn't a control. The gateway or the application enforces it.
- Hitting a budget is a failure mode like an outage, and needs a planned response.
Practise the rest
6 Budgets questions, 492 in all, each with a model answer. Free.