Mid-level interview questionAsked at Decagon · interview write-up
Design an AI gateway in front of several external LLM providers, with routing and fallback, quotas, retries, circuit breakers and privacy-preserving logging.
Model answer
One API; adapters normalize each provider's schema and streaming. Route by model alias, cost and region; fall back only to models evaluated as equivalent.
Quotas: token buckets per tenant, counting requests and tokens. Retry only timeouts, 429s and 5xx, with jittered backoff, honouring Retry-After, under a retry budget. A circuit breaker per provider opens on error rate, then sends probes.
Log metadata always: tokens, latency, cost, status. Bodies are redacted or skipped, with short retention.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Decagon interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?