Topic 3
Routing by intent
Most applications start with one model for everything. As they grow, AI Engineering (Ch. 10, "Router") suggests a router: send different kinds of query to different solutions.
Predict first
Scroll sideways to see the whole diagram
AI Engineering: instead of one model for every query, a router sends each to the best solution. It's usually an intent classifier, and it should be fast and cheap.
Text version
- AI Engineering: instead of one model for every query, a router sends each to the best solution. It's usually an intent classifier, and it should be fast and cheap.
- The book's customer-support example: password resets to the FAQ, billing mistakes to a person, technical issues to a troubleshooting bot.
- Out-of-scope queries get a polite stock response without spending a model call.
- Two benefits: specialised solutions can outperform a general model, and simple queries can go to cheaper models.
The book gives two benefits. Specialised solutions, such as a model tuned for troubleshooting and another for billing, can beat a general model on their own queries. And routing saves money: simple queries can go to cheaper models instead of an expensive one for everything.
A router is usually an intent classifier. The book notes many teams adapt small models (it mentions GPT-2, BERT and Llama 7B), and many train even smaller classifiers from scratch; either way, routers should be fast and cheap so that using several doesn't add much latency or cost. A router can also:
- ask for clarification on ambiguous queries; the book's example is “Freezing”: the account, or the weather?
- adjust the context when models have different limits, truncating or choosing a model with a larger window;
- predict the next action in an agent, as in Module 06.
Quick check