Lesson 3 · 7 topics · 20 min · intermediate

Choosing Context: Selection, Ordering and Compression

You will be able to

Treat the context as a budget: explain how each added token costs money, prefill time and window space.
Select context by retrieval score with top-k and a token budget, and explain how keyword, embedding and fused (RRF) rankings decide what is lost at the cut.
Describe the 'lost in the middle' position effect from Liu et al.'s measurements, and design a needle-in-a-haystack test for your own model.
Order selected passages deliberately (best first or best next to the question) instead of in stored order, and say why inclusion matters more than order.
Compare dropping, truncating and summarising as ways to compress context, and name what each one loses.
Explain why a high similarity score is not the same as relevance, using a real case where both retrievers rank the wrong passage first.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 5 "Prompt Engineering": "Introduction to Prompting" > "Context Length and Context Efficiency". Ch. 6 "RAG and Agents": "RAG"; "Combining retrieval algorithms"; "Retrieval Optimization" > "Reranking"; "Memory". Ch. 10 "AI Engineering Architecture and User Feedback": "Step 3. Add Model Router and Gateway" > "Router".
Lost in the Middle: How Language Models Use Long Contexts · Nelson F. Liu et al., 2023
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang. Transactions of the Association for Computational Linguistics (arXiv:2307.03172v3). Used: Abstract; Section 2 (multi-document QA setup), Figure 5, Table 1; Section 5 "Is More Context Is Always Better?"; Appendix G.2, Table 6 (fetched 2026-10-21). Open source

You will be able to

Treat the context as a budget: explain how each added token costs money, prefill time and window space.

Select context by retrieval score with top-k and a token budget, and explain how keyword, embedding and fused (RRF) rankings decide what is lost at the cut.

Describe the 'lost in the middle' position effect from Liu et al.'s measurements, and design a needle-in-a-haystack test for your own model.

Order selected passages deliberately (best first or best next to the question) instead of in stored order, and say why inclusion matters more than order.

Compare dropping, truncating and summarising as ways to compress context, and name what each one loses.

Explain why a high similarity score is not the same as relevance, using a real case where both retrievers rank the wrong passage first.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 5 "Prompt Engineering": "Introduction to Prompting" > "Context Length and Context Efficiency". Ch. 6 "RAG and Agents": "RAG"; "Combining retrieval algorithms"; "Retrieval Optimization" > "Reranking"; "Memory". Ch. 10 "AI Engineering Architecture and User Feedback": "Step 3. Add Model Router and Gateway" > "Router".

Lost in the Middle: How Language Models Use Long Contexts · Nelson F. Liu et al., 2023

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang. Transactions of the Association for Computational Linguistics (arXiv:2307.03172v3). Used: Abstract; Section 2 (multi-document QA setup), Figure 5, Table 1; Section 5 "Is More Context Is Always Better?"; Appendix G.2, Table 6 (fetched 2026-10-21). Open source