Topic 5

Cascades: cheap first, escalate on failure

A router decides before any model answers. A cascade lets the cheap model answer first and decides afterwards.

FrugalGPT, from Chen, Zaharia and Zou (2023), describes three ways to cut the cost of using language model APIs: adapting prompts, approximating expensive models, and cascades. Its cascade learns which models to try for different queries; on the paper's benchmarks it matched GPT-4's performance at up to 98% lower cost, or beat GPT-4's accuracy by 4% at the same cost. The paper also notes that API prices vary by up to about 100× between providers.

Predict first

What can a cascade use to decide on escalation that a router can't?

The user's nameThe cheap model's actual answer: a scorer or validator can check it before decidingThe time of dayNothing more

A cascadeStep 1 / 5

Scroll sideways to see the whole diagram

A cascade tries the cheap model first. FrugalGPT (Chen et al., 2023) learns which models to try, in which order, for different queries.

Text version

A cascade tries the cheap model first. FrugalGPT (Chen et al., 2023) learns which models to try, in which order, for different queries.
Unlike a router, it decides after seeing an answer, so it can use the answer itself as evidence.
If the answer passes, stop: the expensive model was never called.
If it fails, escalate. FrugalGPT reported matching GPT-4's performance at up to 98% lower cost on its benchmarks.
The price: escalated queries pay twice, plus the check, and they're slower. A cascade wins when most queries stop at the first step and the check is reliable.

In the lab on the previous page, the cascade point gets 85% at 6.25 cost units per query: the small model answers all twenty, and a check sends 7 failed answers to the large model. The check is imperfect: it misses one wrong answer on a borderline query. At the same accuracy the router needs 7.3 units, so here the cascade is cheaper. But the lab treats the check as free; a model-based scorer isn't.

When cascades work well:

Most queries stop at the first step. Every escalation pays for both models.
The check is reliable. Missed failures ship wrong answers; false alarms waste the large model on answers that were fine.
Latency allows it. Escalated queries wait for two models in sequence.

Cheap checks exist for many tasks: a JSON schema, a unit test, a regex, a structured-output validator. Those make cascades especially attractive.

Quick check

Your task produces SQL. Which check makes a good cascade gate?

Asking the small model if it's confidentCounting tokensRunning the query against a test database and checking it executes and returns the expected columnsChecking the query is short