Topic 1

Fine-tuning comes later

Sooner or later someone asks: shouldn't we fine-tune a model on our own data? Sometimes yes. AI Engineering (Ch. 7, "When to Finetune") is clear about the order, though: compared with prompt-based methods, fine-tuning needs significantly more resources (data, hardware, and machine-learning expertise), so it's generally attempted after extensive experiments with prompting. And the two aren't exclusive: real problems often need both.

Predict first

A team says prompting “doesn't work” and wants to fine-tune. What does the book suggest checking first?

Nothing: fine-tuneWhether a bigger GPU is availableWhether the prompt experiments were systematic: clear instructions, representative examples, well-defined metricsWhether competitors fine-tune

Adapting a model to a task, step by stepStep 1 / 5

Scroll sideways to see the whole diagram

AI Engineering: before any adaptation, define your evaluation criteria and pipeline. Every later step is measured with it.

Text version

AI Engineering: before any adaptation, define your evaluation criteria and pipeline. Every later step is measured with it.
Start with prompting alone, then add examples to the prompt: the book suggests anywhere from 1 to 50.
If the model fails for lack of information, connect it to data, starting with basic retrieval such as term-based search (Module 05).
Then follow the failures: information failures call for better retrieval; behaviour failures (irrelevant, malformatted or unsafe outputs) for fine-tuning.
The two can be combined. The book notes better retrieval adds inference complexity, while fine-tuning adds development complexity but leaves inference unchanged.

The book's workflow, which pulls the course together:

Prompting alone, with versioned prompts (Module 03).
More examples in the prompt: the book suggests anywhere from 1 to 50, depending on the task.
Retrieval, if failures come from missing information, starting simple, such as term-based search (Module 05).
Follow the failures: more advanced retrieval for information failures, fine-tuning for behaviour failures.
Combine RAG and fine-tuning where it measurably helps.

And before all of it: evaluation criteria and a pipeline (Module 04), present at every step, not only the first.

Quick check

Why does the book put evaluation before the first adaptation step?

Without it you can't tell whether prompting, retrieval or fine-tuning actually improved anything, or which failures remainTraditionEvaluation is cheapFine-tuning needs it for training