Topic 1

What RAG is

A model only knows what was in its training data and what's in its prompt. Retrieval-augmented generation (RAG) adds a step before generation: search your own data for the passages most relevant to the question, put them in the prompt, and have the model answer from them.

The pattern is older than the name. AI Engineering (Ch. 6, "RAG") traces it to Chen et al. (2017), "Reading Wikipedia to Answer Open-Domain Questions": the system retrieved the five Wikipedia pages most relevant to a question, and a reader model pulled the answer out of them. (The paper's retriever used bigram hashing and TF-IDF.) Lewis et al. (2020) coined the term retrieval-augmented generation, for knowledge-intensive tasks where all the knowledge can't be given to the model directly.

Predict first

Your support assistant answers from a model trained a year ago. The pricing page changed last week. What does RAG change?

Nothing: the model still only knows its training dataThe model is retrained on the new pageThe answer can use this week's pricing page, because the page is retrieved at question time and put in the promptThe model's weights are updated with the new price

Retrieval-augmented generationStep 1 / 4

Scroll sideways to see the whole diagram

Search your own documents for passages related to the question, with keywords, embeddings or both.

Text version

Search your own documents for passages related to the question, with keywords, embeddings or both.
A reranker reads the question with each candidate and keeps the few that actually answer it.
The kept passages go into the prompt, with instructions to answer only from them.
The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources.

The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems.

AI Engineering describes RAG as building context specific to each query, instead of using the same context for every query. That gives three things a bigger model alone doesn't:

Fresh knowledge. Updating a document updates what the model can draw on. Lewis et al. list updating a model's knowledge as an open problem for models that store knowledge only in their weights.
Per-user data. A query about one customer can include that customer's records and nobody else's.
Checkable answers. Retrieved passages can be cited, so a reader, or an evaluator, can check the answer against its source. Lesson 5 measures this.

The book also compares context construction to feature engineering in classical machine learning: both give the model the information it needs to handle an input.

Quick check

Which is a reason to use RAG rather than relying on a model's training data?

It makes the model fasterIt removes the need for a promptAnswers can draw on current, per-user data, and can cite where they came fromIt guarantees the answer is correct