Topic 2

Retrieve, or put it all in the prompt?

If context windowsContext windowThe maximum number of tokens a model can process at once: prompt and output together.Where this is taught keep growing, why not skip retrieval and put the whole knowledge base in every prompt? Sometimes that's the right call.

Anthropic's post on contextual retrieval (2024) says that if a knowledge base is smaller than 200,000 tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught, about 500 pages, you can include all of it in the prompt with no need for RAG. AI Engineering (Ch. 6, "RAG") quotes that guidance and gives two reasons RAG doesn't go away as windows grow:

1. Data outgrows windows. Organisations add data far more often than they delete it, so some applications will always have more than fits.
2. A long context isn't always used well. The more context, the more likely the model attends to the wrong part, and every extra token costs money and may add latency. You saw this in lost in the middle.

Move the knowledge base past 200,000 tokens and the choice is made for you. Below it, the trade-off is per-query cost and how well the model uses a long context, against the work of building and evaluating a retriever. A small, stable corpus is a good case for putting it all in; a large or fast-changing one needs retrieval.