Hybrid search and reranking interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 hybrid search and reranking questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Hybrid Search and Reranking teaches the ideas behind them.Updated
Model answer
Combining term-based and embedding-based retrieval, because they fail differently.
Scroll sideways to see the whole diagram
In sequence: a cheap, less precise retriever fetches a candidate set. AI Engineering's example: every document containing “transformer”.
Text version
- In sequence: a cheap, less precise retriever fetches a candidate set. AI Engineering's example: every document containing “transformer”.
- A more precise, more expensive step orders the candidates: here, vector search picks the ones about the neural architecture. This second step is also called reranking.
- The catch: anything the first step didn't fetch is gone for good.
- In parallel: run several retrievers on the same query, each producing its own ranking.
- Combine the rankings into one, for example with reciprocal rank fusion. A document only one retriever found can still make the final list.
Either in sequence (a cheap retriever fetches candidates, a precise step orders them) or in parallel (both rank, then the rankings are fused).
Model answer
Each retriever gives a document 1 / (k + rank); sum across retrievers. BM25 points and cosine similarities aren't on the same scale, but ranks always are.
k = 60 keeps one retriever's first place from dominating, so agreement wins. Cormack et al. found k = 60 near-optimal but not critical.
Model answer
When one retriever returns noise for a class of queries. On the course's six queries, BM25 matched a single common word (“every”, “phone”) in unrelated articles, those spurious matches got votes, and fusion ranked the right article first for 3 of 6, against 5 of 6 for embeddings alone.
Fusion is only as good as its inputs: measure it on your queries.
Model answer
An embedding model encodes query and document separately, so document vectors are precomputed and search scales to millions.
Scroll sideways to see the whole diagram
An embedding model encodes the query and each document separately.
Text version
- An embedding model encodes the query and each document separately.
- Relevance is a comparison between two vectors that never saw each other's text.
- Because document vectors don't depend on the query, they're computed once. That's what makes searching a large collection fast.
- A reranker reads the query and one candidate together, as a single input, so every word of one can attend to every word of the other. Hands-On LLMs calls this a cross-encoder.
- It outputs a relevance score. Nothing can be precomputed, so it runs once per candidate at query time: affordable for a shortlist, not for the whole collection.
A reranker is typically a cross-encoder: it reads query and candidate together and scores relevance. More precise, but one model run per candidate at query time, so it only sees a shortlist.
Model answer
From where the right chunk usually lands. A reranker can only reorder what it's given: if relevant chunks typically sit at rank 8, a pool of 3 can't help.
Each extra candidate is another model run, so the pool trades quality against latency, as Nogueira et al. note for multi-stage ranking. Measure recall at the pool size, then precision at the top after reranking.
Model answer
A reranker is a better ranker, not a reasoner. On the course's data it ranked the right article first for 5 of 6 queries, but scored the reassuring toy article 4.66 against 3.89 for the one that answers.
So keep more than one candidate, instruct the generator to read carefully and answer only from the passages, and keep negation cases in the evaluation set.