Choosing and routing models interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 choosing and routing models questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Choosing and Routing Models teaches the ideas behind them.Updated
Model answer
Filter on hard attributes you can't change (licence, model size, privacy, control), narrow with public benchmarks, run your own evaluation balancing quality, latency and cost, then monitor in production. You want the best model for your application, not the leaderboard.
Scroll sideways to see the whole diagram
AI Engineering: hard attributes are what you can't or won't change, such as licences, training data, privacy policy, or latency of a model someone else hosts. Filter on them first.
Text version
- AI Engineering: hard attributes are what you can't or won't change, such as licences, training data, privacy policy, or latency of a model someone else hosts. Filter on them first.
- Use public benchmarks and leaderboards to narrow the list, while knowing they can't be fully trusted for your task.
- Then run your own evaluation pipeline, balancing quality, latency and cost.
- Keep monitoring in production, and feed failures back into evaluation.
- The book stresses the steps are iterative: you might plan to self-host, then find open models can't reach the quality you need.
Model answer
The model behind an API name can change, sometimes without notice. The book cites Voiceflow reporting a 10% drop between GPT-3.5-turbo versions.
Pin a dated version where offered, and treat any switch as a release: rerun the evaluation and regression suites.
Model answer
Usually an intent classifier: sends each query to the best solution, a specialised model, a cheaper model, a person, or a stock reply for out-of-scope requests with no model call. It can also ask for clarification. It must be small and fast, since it runs on every query.
Scroll sideways to see the whole diagram
AI Engineering: instead of one model for every query, a router sends each to the best solution. It's usually an intent classifier, and it should be fast and cheap.
Text version
- AI Engineering: instead of one model for every query, a router sends each to the best solution. It's usually an intent classifier, and it should be fast and cheap.
- The book's customer-support example: password resets to the FAQ, billing mistakes to a person, technical issues to a troubleshooting bot.
- Out-of-scope queries get a polite stock response without spending a model call.
- Two benefits: specialised solutions can outperform a general model, and simple queries can go to cheaper models.
Model answer
A router decides before answering, from the question. A cascade lets the cheap model answer, checks the answer, and escalates failures, so it decides with more evidence.
Cascades win when most queries stop at step one and a cheap, reliable check exists: a schema, a test, a validator. Escalated queries pay for both.
Model answer
Trace its cost–accuracy curve on your evaluation set and pick the cheapest point meeting your quality bar. In the course's illustrative example, one threshold matched the large model's 90% at about half its cost; RouteLLM reports over 2× savings in some cases.
Moving the threshold moves the borderline queries, so re-measure, broken down by query type.
Model answer
One interface to every model, so a provider change happens in one place. Plus access control instead of shared provider keys, usage limits and cost tracking, fallbacks when a provider fails, and logging; some add caching and guardrails.