Topic 1
The best model for your application
This module is about running AI applications for real users: choosing and routing models, surviving outages, controlling cost, watching what happens, guarding inputs and outputs, and deciding when to fine-tune. It starts with the first decision: which model.
AI Engineering (Ch. 4, "Model Selection") opens with a point that's easy to forget: you don't care which model is the best. You care which model is best for your application. And you'll make the choice more than once: prompt engineering might start with the strongest model to test feasibility, then work back to see whether smaller models will do.
Predict first
Scroll sideways to see the whole diagram
AI Engineering: hard attributes are what you can't or won't change, such as licences, training data, privacy policy, or latency of a model someone else hosts. Filter on them first.
Text version
- AI Engineering: hard attributes are what you can't or won't change, such as licences, training data, privacy policy, or latency of a model someone else hosts. Filter on them first.
- Use public benchmarks and leaderboards to narrow the list, while knowing they can't be fully trusted for your task.
- Then run your own evaluation pipeline, balancing quality, latency and cost.
- Keep monitoring in production, and feed failures back into evaluation.
- The book stresses the steps are iterative: you might plan to self-host, then find open models can't reach the quality you need.
The book's four steps:
- Filter out models whose hard attributes don't work for you.
- Use public information, benchmarks and leaderboards, to narrow down the promising ones, balancing quality, latency and cost.
- Run your own evaluation to find the best: the pipeline of Module 04.
- Monitor in production to detect failures and collect feedback.
Within the shortlist, the book describes the choice in two parts: work out the best achievable performance, then map models on cost against performance and pick the one that gives the best performance for the money. What counts as hard depends on your situation: latency is soft if you host the model and can optimise it, hard if someone else hosts it.
Quick check