Mid-level interview question
Why is term-based search fast, and what does it never return?
Model answer
An inverted index maps each term to the documents containing it, with counts, so only documents on the query terms' lists are scored.
Scroll sideways to see the whole diagram
The query, after lowercasing and dropping stop words, is three terms that appear in the collection.
Text version
- The query, after lowercasing and dropping stop words, is three terms that appear in the collection.
- An inverted index maps each term to the documents that contain it, with how often: like the index at the back of a book.
- Look up each query term. The number of documents in each list is the term's document frequency, which IDF needs.
- Only documents on some list are scored. That's why term-based search is fast, and why a document sharing no term is never returned.
Terms and articles from the search-lab help centre (lib/labs/data/search-lab.json).
The flip side: a document sharing no term with the query is never a candidate, however well it answers. Paraphrase is term search's blind spot.
Practise the rest
6 Retrieval algorithms questions, 492 in all, each with a model answer. Free.
More Retrieval algorithms interview questions
- What do TF and IDF measure?
- What does BM25 add to TF-IDF?
- How does embedding retrieval work, and what's its catch?
- Exact k-NN is too slow at your scale. What do you give up with an ANN index, and how do you choose one?
- Term-based or embedding retrieval for a support search full of error codes and paraphrased questions?