Topic 2
Licences, and models that change underneath you
One hard choice comes early: call a provider's API, or host an open-weight model yourself. Module 02 compared their costs. AI Engineering (Ch. 4, "Model Build Versus Buy") adds two things that matter in production.
Licences. Open-weight models come under many licences, and the book suggests asking: is commercial use allowed? With what restrictions? Its example: Llama 2 and Llama 3 require a special licence from Meta for applications with more than 700 million monthly active users. And can the model's outputs be used to train other models?
Control over change. With an API, the model is someone else's to update.
Predict first
Scroll sideways to see the whole diagram
With a model API, you don't control the model.
Text version
- With a model API, you don't control the model.
- AI Engineering warns the API can stay the same while the model behind it is updated, not always with notice. Chen et al. (2023) saw notable benchmark differences between the March and June 2023 versions of GPT-4 and GPT-3.5.
- Pinning a dated version, where the provider offers one, makes changes deliberate.
- The book cites Voiceflow reporting a 10% performance drop moving from GPT-3.5-turbo-0301 to GPT-3.5-turbo-1106. Treat a model switch like any release.
- Even pinned, keep monitoring: lesson 4 covers drift detection.
The book also cites Chen et al. (2023), who observed notable differences in benchmark scores between the March 2023 and June 2023 versions of GPT-4 and GPT-3.5. The production habits that follow:
- Pin a version where the provider offers dated or versioned model names, so changes happen when you decide.
- Treat a model switch as a release: rerun the regression suite and the evaluation before switching.
- Watch for drift anyway, which lesson 4 covers.
Self-hosting removes the surprise updates, but not the work: you take on serving, scaling and upgrades yourself.
Quick check