Topic 5
Deciding where to look
The simplest RAG system searches the same index for every message. Real questions don't all need that.
Hands-On Large Language Models (Ch. 8, "Advanced RAG Techniques") lists improvements that hand more of the decision to the model:
- Query routing: choose the source. HR questions go to the HR system; customer questions go to the CRM.
- No search: let the model decide when a question needs no retrieval and can be answered directly.
- Multi-query: split a question that needs several searches ("compare 2023 and 2024 results") into one search each.
- Multi-hop: search, read the result, then search again based on what came back.
The book notes that each step gives the model more responsibility, until the system starts to look like an agent: Module 06.
Predict first
Scroll sideways to see the whole diagram
Before searching, decide whether the question needs a search at all, and which source can answer it.
Text version
- Before searching, decide whether the question needs a search at all, and which source can answer it.
- Some messages need no retrieval: answering them directly saves a search and keeps irrelevant text out of the prompt.
- Questions about how things work go to the document index.
- Questions about numbers in a table (“how many units sold last week?”) need a SQL query written from the schema and executed, not a text search.
- Whatever the route, the generator answers from what came back.
With many tables, the book adds a step before text-to-SQL: predict which tables a query needs, since all their schemas may not fit in the context. The same applies to document collections. Searching the right source is the first retrieval decision, and a wrong route fails before any ranking happens.
Quick check