Budgets, rate limits and usage controls interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 budgets, rate limits and usage controls questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Budgets, Rate Limits and Usage Controls teaches the ideas behind them.Updated
Model answer
Cost, rate and latency. Each is a requirement enforced in code with a planned behaviour when it's hit. A provider's spend cap stops every feature on the account until it resets.
Scroll sideways to see the whole diagram
Cost: what each request, each user and the whole feature may spend. Module 02 showed how to compute it.
Text version
- Cost: what each request, each user and the whole feature may spend. Module 02 showed how to compute it.
- Rate: how many requests and tokens per minute, both the limits a provider imposes on you and the ones you impose on your users.
- Latency: how long a user waits, for the first token and for the whole answer.
- A budget that's only a number in a spreadsheet isn't a control. The gateway or the application enforces it.
- Hitting a budget is a failure mode like an outage, and needs a planned response.
Model answer
Prompt caching of a shared prefix, routing easy requests to a smaller model, shorter outputs, and fewer input tokens. In the course's illustrative model, caching or routing roughly halves a $27,000 month; both together reach about $8,000.
Each has a quality side: measure it before shipping.
Model answer
When the answer depends on who asked. The book's example: a return-policy answer built from user X's membership, cached as generic, then served to user Y.
Scroll sideways to see the whole diagram
AI Engineering: exact caching suits answers that are the same for everyone, and is especially appealing for multi-step or slow work like retrieval, SQL or web search.
Text version
- AI Engineering: exact caching suits answers that are the same for everyone, and is especially appealing for multi-step or slow work like retrieval, SQL or web search.
- User-specific queries (“What's the status of my recent order?”) are unlikely to be reused, and time-sensitive ones go stale. Many teams train a classifier to decide what to cache.
- The book's warning: a return-policy question looks generic, but the answer used user X's membership details. Cached as generic, it reveals X's information to the next user who asks.
- So a cache key must include everything the answer depended on, or the answer must not be cached.
Key the cache on everything the answer depended on, skip user-specific and time-sensitive queries, and invalidate on prompt or data changes.
Model answer
Limits use a token bucket: capacity refills continuously, so short bursts can be rejected even when the per-minute total is fine. Smooth bursts, wait at least retry-after with backoff, separate batch from live traffic, and use prompt caching, which for most Claude models doesn't count toward input-token limits.
Model answer
Nested: provider limits for the organisation; per-application limits in the gateway so one can't starve others; per-user limits against abuse and compromised accounts; per-request caps on tokens and agent steps.
Track spend daily, project the month, and act before the hard cap.
Model answer
Add up the stages before the first token: router, retrieval, reranker, the model's time to first token. Streaming hides generation, but not for answers needed whole, like JSON or output-guardrail checks.
Parallelise independent stages, skip optional ones under load, and budget percentiles, not averages.