Junior interview question
What is graceful degradation?
Model answer
When parts fail or load is too high, return lower-quality, cheaper results instead of nothing: a smaller model, no reranker, a cached answer, the raw passages, or an honest message. Never a hang.
Scroll sideways to see the whole diagram
Normally, the primary model and the full pipeline answer.
Text version
- Normally, the primary model and the full pipeline answer.
- If it's down or rate-limited, a gateway can route to another model. AI Engineering notes API failures are unfortunately common.
- Under load, do less: the SRE book's graceful degradation returns lower-quality, cheaper-to-compute results rather than nothing.
- With no model at all, a cached answer or the raw retrieved passages may still help.
- The bottom rung is an honest message or a person. What's unacceptable is a hang, a blank screen, or an error the user can't act on.
Practise the rest
6 Fallbacks questions, 492 in all, each with a model answer. Free.
More Fallbacks interview questions
- Your primary is 99% available and a fallback 99.5%. What's combined availability?
- What does a circuit breaker do?
- Demand is twice capacity. Why shed load instead of queueing everything?
- You fall back to a model you've never evaluated with your prompt. What can go wrong?
- Retry after a bad output, or send two requests in parallel?