Topic 7
Term or embedding? And wrap-up
Which family should you use? AI Engineering (Ch. 6, "Comparing retrieval algorithms") compares them on speed, performance and cost:
- Speed. Term-based is generally much faster, both indexing and querying: extracting terms is cheaper than computing embeddings, and an index lookup is cheaper than a nearest-neighbour search.
- Performance. Term-based works well out of the box but has few knobs to improve it. Embedding-based retrieval can be improved over time, for example by fine-tuning the embedding model, until it beats term-based, and it handles natural, paraphrased queries.
- Cost. Embeddings cost money to generate and regenerate when data changes, and vector storage and search can be expensive. The book says it's not uncommon for a company's vector database spending to be a fifth to half of what it spends on model APIs.
Predict first
Where does each retriever rank the right article?
BM25 computed live · cosine similarity measured with all-MiniLM-L6-v2 (sentence-transformers 6.1.0)
| Query | BM25 | Embeddings | Similarity to the answer |
|---|---|---|---|
| my internet keeps cutting out | #2 | #1 | 0.418 |
| error 0x80070005 | #1 | #1 | 0.684 |
| the bank charged me a fee for going negative | #1 | #1 | 0.562 |
| how do I stop paying every month | not returned | #1 | 0.575 |
| is this toy safe for a two-year-old | #2 | #2 | 0.708 |
| my phone dies really quickly | not returned | #1 | 0.449 |
The book's warning about embeddings: they can blur exact keywords such as error codes (it uses EADDRNOTAVAIL (99)) or product names. Here MiniLM still finds the 0x80070005 article, but that's the case where term matching is most reliable and embeddings are least. Neither family handles negation: both rank “safe for children over three” above “not safe for children under three”.
Different strengths suggest using both. That's hybrid search, the next lesson.
Wrap-up. From this lesson, you can now:
- compute TF, IDF and a TF-IDF score, and say why rare terms weigh more;
- explain BM25's saturation and length normalisation, and read a score term by term;
- explain how an inverted index makes term search fast, and why non-matching documents are never returned;
- see how term tokenization creates and destroys matches;
- describe embedding retrieval as k-NN over vectors from one model, and why it always returns something;
- name the main ANN index types and the recall, speed and build-time trade-offs;
- compare the two families on speed, quality and cost.
Quick check