Mid-level interview question
Your pipeline concatenates retrieved passages in database order. Does it matter where the most relevant one lands?
Model answer
It can. In Liu et al., GPT-3.5-Turbo with 20 passages scored 75.8% with the answer first, 53.8% at position 10, below its 56.1% closed-book score, and 63.2% last. Claude-1.3 was much flatter, and the best end varied by model. These are 2023 models, so sort by relevance (best first, or best next to the question) and compare both on a needle-style test.
Practise the rest
12 Context selection questions, 492 in all, each with a model answer. Free.
More Context selection interview questions
- “Our model has a 200K-token window, so let's just send the whole knowledge base.” What's wrong with that?
- You rank passages with reciprocal rank fusion of BM25 and embeddings, and keep the top one. The passage that answers the query never reaches the model. How?
- A long support conversation no longer fits the context window. Compare the ways to shrink it, and how each one fails.
- Your retriever's top passage is usually on topic but often contradicts the right answer. What's going on, and what do you change?
- What is context engineering, and how is it different from prompt engineering?
- Design LLM search or RAG for very long documents and very long user queries.