Mid-level interview questionAsked at Sierra · interview write-up
Design an LLM-powered customer service chat that keeps working when the external LLM provider is slow, fails or returns bad output.
Model answer
Assume the provider will be slow, down or wrong. Set timeouts on time to first token, retry with backoff and jitter only on safe errors, and use a circuit breaker to fail over to another provider or smaller model. Validate each output against a schema and policy checks before showing it; on failure, retry once, then use a scripted reply or human handoff. Keep state server-side so failover resumes cleanly.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Sierra interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?