Topic 3

Form or facts: fine-tuning or RAG

Once prompting is exhausted, AI Engineering (Ch. 7, "Finetuning and RAG") says the choice between RAG and fine-tuning depends on what kind of failures you see.

Information-based failures: the output is factually wrong or outdated, because the model doesn't have the information (it's private to you) or has an old version of it (it changed after the model's training cut-off). RAG gives the model the missing facts.
Behaviour-based failures: the facts may be right, but the output is irrelevant to the task, ignores the expected format, or uses the wrong syntax or style. Fine-tuning can teach the behaviour.

The book's summary: fine-tuning is for form, and RAG is for facts.

Predict first

The book cites Ovadia et al. (2024) on questions about current events. For Mistral-7B, which scored highest: the base model, the base model with RAG, or fine-tuned versions?

Fine-tuned versionsThe base model with RAG, by a wide marginThe base model aloneThey were about equal

Twelve failures from error analysis

Illustrative notes · sorted the way AI Engineering suggests: information or behaviour?

Quoted last year's price for the Pro plan
Didn't know about the March policy change
Invented a feature for an internal tool it's never seen
Answered from memory about a customer's contract
Gave the wrong office hours for the Berlin team
Wrote our query DSL with invalid syntax
Ignored the required report section headings
Spec was accurate but missing the detail engineers need
Used a casual tone in legal summaries
Produced YAML when the schema needed a nested list
Didn't know the product's error codes
Summarised the wrong version of the handbook

Information failures: the model lacks the facts (missing or outdated). Behaviour failures: the facts may be fine, but the form, format, style or relevance isn't.

Sort the twelve illustrative failures. Seven are information failures: prices, policies, error codes the model was never told about. Five are behaviour failures: a query language written with invalid syntax, ignored section headings, the wrong tone. With both kinds, the book's advice is to start with RAG: it's typically easier, with no training data to curate and no fine-tuned model to host, and it should be simple at first (BM25 before vector databases). Then see which behaviour failures remain.

The book also notes the two combine: in the same study, adding RAG on top of a fine-tuned model improved MMLU performance 43% of the time, but not the other 57%. Combining is an experiment too.

Quick check

Your assistant writes correct facts in your company's internal query language, but the syntax is wrong in 30% of answers. Which failure is it, and what's the likely fix?

Behaviour: the facts are fine but the form isn't; constrained outputs or fine-tuning on correct examplesInformation: add RAGNeither: use a bigger contextInformation: fine-tune on the docs