Topic 1
Term frequency and inverse document frequency
Every retriever ranks documents by a relevance score; AI Engineering (Ch. 6, "Retrieval Algorithms") notes that retrieval algorithms differ in how they compute it. This lesson covers the two main families: term-based (also called lexical) and embedding-based (semantic). You'll also see them called sparse and dense; the book avoids that split because sparse embeddings such as SPLADE behave like dense ones. Earlier you saw what each gets right and wrong; here is how each one works.
The simplest term-based idea: retrieve documents that contain the query's words. The book points out two problems. Many documents may contain them, and some query words matter far more than others.
Predict first
Scroll sideways to see the whole diagram
A query is a handful of terms, and some matter more than others: “vietnamese” tells you more than “for”.
Text version
- A query is a handful of terms, and some matter more than others: “vietnamese” tells you more than “for”.
- Term frequency: a document that mentions the term more often is assumed to be more about it.
- Inverse document frequency: a term found in few documents is informative; one found in nearly all is not.
- Each query term contributes its IDF times its frequency in the document.
- The contributions are summed. BM25 keeps this shape and adjusts the frequency part.
Two measurements, as the book defines them:
- Term frequency (TF): how many times a term appears in a document. More mentions suggest the document is more about it.
- Inverse document frequency (IDF): the total number of documents, divided by the number that contain the term. With 10 documents and 5 containing the term, IDF is 10 / 5 = 2. In practice you take the log of that ratio, so very rare terms don't swamp everything else.
TF-IDF sums IDF × TF over the query's terms. It ranks a document higher the more often it uses the query's rare words.
Quick check