Mid-level interview question
You rank passages with reciprocal rank fusion of BM25 and embeddings, and keep the top one. The passage that answers the query never reaches the model. How?
Model answer
Fusion rewards agreement, not correctness: each retriever gives 1 / (k + rank), k = 60 being typical.
Scroll sideways to see the whole diagram
Retrievers propose candidates. BM25 returns only passages that share a word with the query; embedding search always returns a nearest neighbour.
Text version
- Retrievers propose candidates. BM25 returns only passages that share a word with the query; embedding search always returns a nearest neighbour.
- Candidates are ranked by one retriever's score, or by fusing the two rankings (reciprocal rank fusion).
- Selection is a cut: the top k that fit the budget go in. A good passage ranked below the cut is lost, however relevant.
- Then the order. Ranking for a search page and ordering for a model are different jobs: the model reads all of it, but not equally well everywhere.
After AI Engineering Ch. 6, "RAG": "Combining retrieval algorithms" and "Reranking"; Liu et al. 2023 (arXiv:2307.03172), Section 5.
For "how do I stop paying every month", BM25 returned only the Wi-Fi article, through the word "every"; embeddings ranked it third. It scored 1/61 + 1/63 and beat the cancellation article's 1/61, so the answer, ranked #2, was cut. Judge selection on queries with known answers, and leave k room.
Practise the rest
12 Context selection questions, 492 in all, each with a model answer. Free.
More Context selection interview questions
- “Our model has a 200K-token window, so let's just send the whole knowledge base.” What's wrong with that?
- Your pipeline concatenates retrieved passages in database order. Does it matter where the most relevant one lands?
- A long support conversation no longer fits the context window. Compare the ways to shrink it, and how each one fails.
- Your retriever's top passage is usually on topic but often contradicts the right answer. What's going on, and what do you change?
- What is context engineering, and how is it different from prompt engineering?
- Design LLM search or RAG for very long documents and very long user queries.