Topic 4
Reasons not to
AI Engineering (Ch. 7, "Reasons Not to Finetune") starts by noting that many of fine-tuning's improvements can be achieved, to some extent, without it: better prompts and context also improve quality, and other techniques also produce structured outputs. Then it lists what fine-tuning costs.
Predict first
Scroll sideways to see the whole diagram
AI Engineering's list of up-front costs: data, which can be slow and expensive to annotate, and the knowledge to train models well.
Text version
- AI Engineering's list of up-front costs: data, which can be slow and expensive to annotate, and the knowledge to train models well.
- Then serving the fine-tuned model, and a budget for maintaining it while new base models keep arriving, possibly better than yours.
- Fine-tuning for one task can degrade performance on others, a problem when one model serves several kinds of request.
- The book's cautionary tale: BloombergGPT, a 50B finance model trained for an estimated $1.3–2.6 million in compute, was outperformed on financial benchmarks by GPT-4-0314 released the same month (Li et al., 2023).
AI Engineering (Huyen), Ch. 7, Table 7-1 and "Finetuning domain-specific tasks", citing Wu et al. (2023) and Li et al. (2023).
The other costs the book lists:
- Data: annotated examples can be slow and expensive, especially for tasks needing expertise.
- Expertise: choosing a base model, training knobs, overfitting, evaluating during training.
- Serving: hosting your own model, or a provider's service, with Module 02's inference work.
- Maintenance: new base models arrive fast and may improve faster than your fine-tuned one. When does a new base model justify switching, or re-fine-tuning?
The book's cautionary example is BloombergGPT: a 50-billion-parameter model for finance, trained with about 1.3 million A100 GPU hours, an estimated $1.3–2.6 million in compute. In the same month, GPT-4-0314 was released, and research by Li et al. (2023) found it significantly outperformed BloombergGPT on financial benchmarks: 87.15 against 75.07 weighted F1 on FiQA sentiment analysis, and 76.48 against 43.41 accuracy on ConvFinQA, zero-shot. The book is careful to add that benchmarks don't capture real-world use, and that the team gained valuable experience.
Quick check