Topic 5
Embedding-based retrieval
Term-based retrieval scores spelling. AI Engineering (Ch. 6, "Embedding-based retrieval") gives the classic failure: a query for “transformer architecture” can return documents about electrical transformers, or the Transformers films. Embedding-based retrieval ranks documents by how close their meaning is to the query's, using the text embeddings of Module 01.
Predict first
Scroll sideways to see the whole diagram
Indexing gains a step: every chunk is turned into an embedding and stored in a vector database.
Text version
- Indexing gains a step: every chunk is turned into an embedding and stored in a vector database.
- At query time, embed the query with the same embedding model, and compare it with the stored vectors.
- Return the k closest. Comparing against every vector is exact k-nearest-neighbour search: fine for small collections, slow for large ones.
- Unlike term search, this always returns k results. A minimum similarity can filter out weak matches, but thresholds are model-specific.
The book describes the query in two steps: embed the query with the same model used during indexing, then fetch the k chunks whose embeddings are closest. The naive way is k-nearest neighbours (k-NN): compute the query's similarity to every stored vector, sort, return the top k. It's exact, and the book recommends it only for small datasets because it's slow on large ones.
One behaviour to design for: embedding search always returns k results. Hands-On Large Language Models (Ch. 8, "Caveats of dense retrieval") points out that a query with no answer in the collection still gets results with distances, and suggests a maximum-distance threshold as one heuristic. The book also warns that dense retrieval can struggle in domains its model wasn't trained on: a model trained on web text may do poorly on legal documents.
Quick check