BM25
A keyword-search scoring function that ranks documents by the query words they contain, weighting rare words more and long documents less.
Interview questions about BM25
- You rank passages with reciprocal rank fusion of BM25 and embeddings, and keep the top one. The passage that answers the query never reaches the model. How?
Fusion rewards agreement, not correctness: each retriever gives 1 / (k + rank), k = 60 being typical.
- What does BM25 add to TF-IDF?
Two corrections to term frequency: - Saturation: the tenth mention adds less than the first (parameter k1).
- Your retriever's top passage is usually on topic but often contradicts the right answer. What's going on, and what do you change?
Retrieval scores similarity, not whether a passage answers. For "is this toy safe for a two-year-old", BM25, embeddings and their fusion all put "safe for…
- How does reciprocal rank fusion work, and why use ranks rather than scores?
Each retriever gives a document 1 / (k + rank); sum across retrievers. BM25 points and cosine similarities aren't on the same scale, but ranks always are.
- When can fusion make retrieval worse?
When one retriever returns noise for a class of queries. On the course's six queries, BM25 matched a single common word (“every”, “phone”) in unrelated…
- Anthropic reports large gains from contextual retrieval. What exactly did they measure, and what does it cost?
1 − recall@20: the share of relevant chunks missing from the top 20.
- Why combine keyword and embedding search, and where does a reranker fit?
Embeddings catch paraphrases; keyword search such as BM25 catches exact terms like product codes, names and error messages.
- Design a RAG system that has to meet strict latency and cost targets while serving many tenants.
Start by splitting the latency budget across retrieval, reranking and generation.