Topic 3
Indexing, then querying
A RAG system has two components, AI Engineering (Ch. 6, "RAG Architecture") explains: a retriever that finds information, and a generator that answers from it. The retriever does two jobs at different times:
- Indexing, ahead of time: process the data so it can be found quickly later.
- Querying, per request: send the question and get back the most relevant pieces.
Predict first
Scroll sideways to see the whole diagram
Indexing happens ahead of time. Documents are split into chunks small enough to put in a prompt.
Text version
- Indexing happens ahead of time. Documents are split into chunks small enough to put in a prompt.
- Each chunk is indexed for the kind of search you'll run later: its terms, its embedding, or both.
- Querying happens per request: the question is matched against the index and the top k chunks come back.
- The chunks are joined with the instructions and the question into one prompt, and the generator answers from it.
Documents vary from ten tokens to a million, so retrieving whole documents can make the context arbitrarily long. Documents are split into chunks first, and the book uses "document" for both a document and a chunk, as information retrieval does. Lesson 4 covers how to chunk.
The original RAG paper trained the retriever and the generator together. Today the two are usually built separately from off-the-shelf parts, though the book notes that fine-tuning the whole system end to end can improve it significantly. Either way, the generator can only use what the retriever hands it.
Quick check