Lesson 3 · 8 topics · 20 min · beginner

Embeddings: From Token IDs to Vectors

You will be able to

Explain an embedding lookup as selecting a row of a learned vocabulary × dimension matrix, and estimate the table's parameter count.
Compute and compare dot product, cosine similarity and Euclidean distance, and explain why normalized vectors make them agree.
Describe how embeddings acquire meaning from context, from word2vec's skip-gram with negative sampling to contrastive training of sentence embedding models.
Distinguish static token embeddings, contextual token embeddings and pooled text embeddings.
Predict where embedding similarity fails: negation, reversed facts, uncalibrated thresholds, truncation, exact strings, and other languages.
Apply embeddings to search, clustering and deduplication while keeping queries and documents in the same model's space.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 2 "Tokens and Embeddings", sections "Token Embeddings", "A Language Model Holds Embeddings for the Vocabulary of Its Tokenizer", "Creating Contextualized Word Embeddings with Language Models", "Text Embeddings (for Sentences and Whole Documents)", "The Word2vec Algorithm and Contrastive Training"; Ch. 8 "Semantic Search and Retrieval-Augmented Generation", section "Caveats of dense retrieval"; Ch. 10 "Creating Text Embedding Models", sections "Embedding Models", "SBERT".
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 3 "Evaluation Methodology", sections "Semantic similarity", "Introduction to Embedding"; Ch. 6 "RAG and Agents", sections "Embedding-based retrieval", "Comparing retrieval algorithms"; Ch. 8 "Dataset Engineering", section "Deduplicate Data".
Speech and Language Processing, 3rd ed. draft · Daniel Jurafsky et al.
Daniel Jurafsky, James H. Martin. Used: Ch. 5 "Embeddings", chapter introduction (draft of August 19, 2026). Open source
Efficient Estimation of Word Representations in Vector Space · Tomas Mikolov et al., 2013
Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. arXiv:1301.3781. Open source
Distributed Representations of Words and Phrases and their Compositionality · Tomas Mikolov et al., 2013
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean. arXiv:1310.4546. Open source
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · Nils Reimers et al., 2019
Nils Reimers, Iryna Gurevych. EMNLP 2019 (arXiv:1908.10084). Open source
all-MiniLM-L6-v2 (model card) · sentence-transformers
sentence-transformers. Open source

You will be able to

Explain an embedding lookup as selecting a row of a learned vocabulary × dimension matrix, and estimate the table's parameter count.

Compute and compare dot product, cosine similarity and Euclidean distance, and explain why normalized vectors make them agree.

Describe how embeddings acquire meaning from context, from word2vec's skip-gram with negative sampling to contrastive training of sentence embedding models.

Distinguish static token embeddings, contextual token embeddings and pooled text embeddings.

Predict where embedding similarity fails: negation, reversed facts, uncalibrated thresholds, truncation, exact strings, and other languages.

Apply embeddings to search, clustering and deduplication while keeping queries and documents in the same model's space.

Sources

Hands-On Large Language Models · Jay Alammar et al., 2024

Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 2 "Tokens and Embeddings", sections "Token Embeddings", "A Language Model Holds Embeddings for the Vocabulary of Its Tokenizer", "Creating Contextualized Word Embeddings with Language Models", "Text Embeddings (for Sentences and Whole Documents)", "The Word2vec Algorithm and Contrastive Training"; Ch. 8 "Semantic Search and Retrieval-Augmented Generation", section "Caveats of dense retrieval"; Ch. 10 "Creating Text Embedding Models", sections "Embedding Models", "SBERT".

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 3 "Evaluation Methodology", sections "Semantic similarity", "Introduction to Embedding"; Ch. 6 "RAG and Agents", sections "Embedding-based retrieval", "Comparing retrieval algorithms"; Ch. 8 "Dataset Engineering", section "Deduplicate Data".

Speech and Language Processing, 3rd ed. draft · Daniel Jurafsky et al.

Daniel Jurafsky, James H. Martin. Used: Ch. 5 "Embeddings", chapter introduction (draft of August 19, 2026). Open source

Efficient Estimation of Word Representations in Vector Space · Tomas Mikolov et al., 2013

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. arXiv:1301.3781. Open source

Distributed Representations of Words and Phrases and their Compositionality · Tomas Mikolov et al., 2013

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean. arXiv:1310.4546. Open source

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · Nils Reimers et al., 2019

Nils Reimers, Iryna Gurevych. EMNLP 2019 (arXiv:1908.10084). Open source

all-MiniLM-L6-v2 (model card) · sentence-transformers

sentence-transformers. Open source