Topic 4

Routing by difficulty

Intent decides what kind of query it is. A second kind of router decides how hard it is, and sends only the hard ones to the expensive model. RouteLLM, from Ong et al. (2024), trains routers that choose between a stronger and a weaker model for each query, learning from human preference data; the paper reports cost reductions of over 2× in some cases without compromising response quality.

The lab makes the trade-off concrete with twenty illustrative queries. The small model answers correctly up to a certain difficulty, the large one up to a higher difficulty, and the large model costs 15 times as much per query. The router doesn't know each query's true difficulty, only a noisy estimate.

Predict first

Sending everything to the large model gets 90% right at 15 cost units per query. Can a router match 90% for less?

No: only the large model gets 90%Yes: at a threshold of 0.5 the router sends half the queries to the large model and still gets 90%, at 8 units per queryYes, at 1 unit per queryOnly by using a third model

Route each query, or use one model?

Illustrative · 20 queries · large model costs 15× the small one

Router threshold (send to the large model above): 0.50

Router accuracy

90%

Cost per query

8.00

Sent to the large model

50%

vs all-large cost

53%

All small: 60% at 1.00 per query
All large: 90% at 15.00 per query
Cascade: 85% at 6.25 per query

Router: estimates difficulty before answering, sends hard-looking queries to the large model. Cascade: the small model answers first; a check sends failed answers on to the large model. Its check is imperfect, and treated as free here.

The curve is the router's whole range: a low threshold sends almost everything to the large model, a high one almost nothing. Read it like a comparison of systems: which point gives the accuracy you need at the lowest cost?

Two cautions, both from the lab's shape:

The router's errors set the curve. With a perfect difficulty estimate, every point would be better. Routers are evaluated like any classifier, on your own queries.
Accuracy can fall faster than cost. Past 0.5, every step saves cost and loses accuracy. Where to stop is a product decision, not a technical one.

Quick check

Your router sends 50% of traffic to the large model. You raise the threshold to cut costs. What must you check?

Only the billNothing: routers don't affect qualityThe router's latency onlyAccuracy on a representative test set at the new threshold, broken down by query type, since the queries newly sent to the small model are the borderline ones