Topic 1
Failure behaviour is a feature
Model APIs go down. They rate-limit you at peak times. They slow down. AI Engineering (Ch. 10, "Gateway") calls API failures “unfortunately common”. A production system doesn't get to choose whether it meets these failures, only how it behaves when it does.
Predict first
Scroll sideways to see the whole diagram
Normally, the primary model and the full pipeline answer.
Text version
- Normally, the primary model and the full pipeline answer.
- If it's down or rate-limited, a gateway can route to another model. AI Engineering notes API failures are unfortunately common.
- Under load, do less: the SRE book's graceful degradation returns lower-quality, cheaper-to-compute results rather than nothing.
- With no model at all, a cached answer or the raw retrieved passages may still help.
- The bottom rung is an honest message or a person. What's unacceptable is a hang, a blank screen, or an error the user can't act on.
The design idea is a ladder: an ordered list of things to try, each a bit worse than the last, ending in something honest rather than a hang. AI Engineering's gateway section describes the first rungs: when the primary API is unavailable, route to alternative models, retry after a short wait, or handle the failure gracefully in other ways. Google's SRE book, "Addressing Cascading Failures", describes the middle rungs as graceful degradation: reduce the work done by returning lower-quality, cheaper-to-compute results.
Which rungs exist, and in what order, is a product decision. A coding assistant might fall back to a smaller model; a medical summariser might prefer to say “unavailable” rather than risk a weaker model's answer.
Quick check