Topic 2

Fallback chains

The first rung below the primary is usually another model: a second provider, a different region, or a self-hosted model. A gateway (lesson 1) makes the switch in one place. How much does it buy?

Predict first

The primary is available 99% of the time; the fallback 99.5%. If their outages are independent, roughly how many minutes a month are both down at once?

About 430About 220About 2Zero

Where do requests end up?

Illustrative availabilities · a 30-day month

Primary available: 99.0%Fallback available: 99.5%Shared outages: 0%

Primary model99.000%
Fallback model0.995%
Degraded answer0.000%
Error0.005%

A model answers

99.995%

Minutes a month with no model

2.2

Primary alone: minutes down

432

Correlation: how often the fallback is down at the same moment as the primary for the same reason (same provider, same region, same upstream). Independent failures multiply; shared ones don't.

Now drag “shared outages” up. If half the primary's outages also take down the fallback (the same provider's two models, the same cloud region, the same upstream dependency), the gap with no model grows to about 217 minutes. If they always fail together, the fallback adds nothing.

That's the most important property of a fallback: it must fail for different reasons. A different provider, a different region, or your own self-hosted model fails independently in a way a second model from the same API often doesn't.

Then switch on the last resort. A cached or static answer is always available, so nothing ends as an error, though some requests get a worse answer. The chain doesn't remove failure; it turns errors into degraded responses.

Quick check

Your fallback is the same provider's smaller model. During the provider's last outage, both failed. What's the fix?

Add a third model from the same providerIncrease timeoutsNothing can be doneUse a fallback that fails independently: another provider, another region, or a self-hosted model