You will be able to
- Design a ladder of fallbacks that ends in an honest response rather than a hang.
- Compute the availability a fallback model adds, and explain why its failures must be independent of the primary's.
- Use circuit breakers so an outage doesn't cost every user a timeout.
- Degrade gracefully under load, and shed load with a fast, clear refusal.
- Evaluate fallback models like any model change, and choose between retrying and parallel requests.
- Write the failure behaviour down and exercise it regularly.
Topics
Your review
Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.
Sign in freeSources
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 10 "AI Engineering Architecture and User Feedback": "Step 2. Put in Guardrails" > "Output guardrails"; "Step 3. Add Model Router and Gateway" > "Gateway".
Site Reliability Engineering · Betsy Beyer et al., 2016
Betsy Beyer, Chris Jones, Jennifer Petoff & Niall Richard Murphy (eds.). O'Reilly. Used: Ch. 22 "Addressing Cascading Failures": "Load Shedding and Graceful Degradation" (fetched 2026-10-23). Open source
CircuitBreaker · Martin Fowler, 2014
Martin Fowler. Used: Fetched 2026-10-23. Open source