Topic 5

A fallback is a different model

A fallback model is a different model. It may format answers differently, follow instructions differently, refuse different things, and need a different prompt. Switching to it silently can turn an outage into a quality incident.

Predict first

Your primary model returns well-formed JSON 99.8% of the time with your prompt. During an outage you fall back to a model you've never evaluated with that prompt. What should you expect?

The same 99.8%Better resultsUnknown: its format reliability, quality and refusals have to be measured with that prompt before you rely on itIt will refuse everything

So the fallback path needs the same treatment as the main one: its own prompt version if needed, its own run of the evaluation and regression suite, and its own metrics in production, broken down by which model answered.

Some failures aren't outages at all but bad outputs: empty, malformed, or failing a guardrail. AI Engineering (Ch. 10, "Output guardrails") notes that many of these can be fixed by retrying, because models are probabilistic, and describes the trade-off:

Retry after, or ask twice at once?Step 1 / 4

Scroll sideways to see the whole diagram

AI Engineering: many output failures (empty, malformed) can be fixed by retrying, because models are probabilistic. But a retry after a failure can double the latency the user sees.

Text version

AI Engineering: many output failures (empty, malformed) can be fixed by retrying, because models are probabilistic. But a retry after a failure can double the latency the user sees.
To keep latency down, send the query twice at once and pick the better response. That costs redundant calls on every request.
Which one fits depends on whether the latency budget or the cost budget is tighter: lesson 3.
The book also describes falling back to people: transferring conversations that contain certain phrases, that a sentiment model flags as angry, or that run past a number of turns.

Retry after a failure: cheap, but the user may wait twice as long.
Ask twice in parallel and pick the better response: latency close to one call, but redundant calls on every request.

The book also describes falling back to people: transferring conversations containing certain phrases to human operators, using a model to decide when to transfer (one team does it when a sentiment model detects anger), or transferring after a certain number of turns so users don't get stuck in a loop.

Quick check

Your latency budget is tight, cost is not, and 5% of responses fail a format check. Which approach fits?

Retry after failureNever retryFall back to a personSend two requests in parallel and use the first valid one