Mid-level interview question
What does a router in front of your models do?
Model answer
Usually an intent classifier: sends each query to the best solution, a specialised model, a cheaper model, a person, or a stock reply for out-of-scope requests with no model call. It can also ask for clarification. It must be small and fast, since it runs on every query.
Scroll sideways to see the whole diagram
AI Engineering: instead of one model for every query, a router sends each to the best solution. It's usually an intent classifier, and it should be fast and cheap.
Text version
- AI Engineering: instead of one model for every query, a router sends each to the best solution. It's usually an intent classifier, and it should be fast and cheap.
- The book's customer-support example: password resets to the FAQ, billing mistakes to a person, technical issues to a troubleshooting bot.
- Out-of-scope queries get a polite stock response without spending a model call.
- Two benefits: specialised solutions can outperform a general model, and simple queries can go to cheaper models.
Practise the rest
6 Model routing questions, 492 in all, each with a model answer. Free.