Lesson 1 · 6 topics · 20 min · intermediate

Choosing and Routing Models

You will be able to

Choose a model by filtering on hard attributes, narrowing with public information, evaluating on your own data, and monitoring.
Account for model licences and for API models that change underneath you; pin versions and treat a model switch as a release.
Route queries by intent, including cheap stock replies for out-of-scope queries.
Route by estimated difficulty between a small and a large model, and read the cost–accuracy curve a router traces.
Use a cascade when a reliable, cheap check on the cheap model's answer exists.
Put a model gateway between applications and models for access control, cost limits, fallbacks and logging.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 4 "Evaluate AI Systems": "Model Selection", "Model Selection Workflow", "Model Build Versus Buy" > "Open source, open weight, and model licenses". Ch. 10 "AI Engineering Architecture and User Feedback": "Step 3. Add Model Router and Gateway" > "Router", "Gateway"; "Drift detection".
RouteLLM: Learning to Route LLMs with Preference Data · Isaac Ong et al., 2024
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous & Ion Stoica. arXiv:2406.18665. Used: Abstract (fetched 2026-10-23). Open source
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance · Lingjiao Chen et al., 2023
Lingjiao Chen, Matei Zaharia & James Zou. arXiv:2305.05176. Used: Abstract (fetched 2026-10-23). Open source

You will be able to

Choose a model by filtering on hard attributes, narrowing with public information, evaluating on your own data, and monitoring.

Account for model licences and for API models that change underneath you; pin versions and treat a model switch as a release.

Route queries by intent, including cheap stock replies for out-of-scope queries.

Route by estimated difficulty between a small and a large model, and read the cost–accuracy curve a router traces.

Use a cascade when a reliable, cheap check on the cheap model's answer exists.

Put a model gateway between applications and models for access control, cost limits, fallbacks and logging.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 4 "Evaluate AI Systems": "Model Selection", "Model Selection Workflow", "Model Build Versus Buy" > "Open source, open weight, and model licenses". Ch. 10 "AI Engineering Architecture and User Feedback": "Step 3. Add Model Router and Gateway" > "Router", "Gateway"; "Drift detection".

RouteLLM: Learning to Route LLMs with Preference Data · Isaac Ong et al., 2024

Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous & Ion Stoica. arXiv:2406.18665. Used: Abstract (fetched 2026-10-23). Open source

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance · Lingjiao Chen et al., 2023

Lingjiao Chen, Matei Zaharia & James Zou. arXiv:2305.05176. Used: Abstract (fetched 2026-10-23). Open source