Topic 1
From token IDs to vectors
The last lesson ended with a list of integers. An integer like 2924 says nothing about meaning on its own: ID 2924 is not "more" than ID 2923, and nearby IDs are not related words. The model's first job is to swap each ID for something it can compute with.
That something is a row of a big learned matrix called the embedding table (or embedding matrix). It has one row per vocabulary entry and a fixed number of columns, the embedding dimension:
Scroll sideways to see the whole diagram →
MiniLM's tokenizer lowercases and splits “the river bank”, then adds two special tokens, [CLS] and [SEP].
Text version
- MiniLM's tokenizer lowercases and splits “the river bank”, then adds two special tokens, [CLS] and [SEP].
- Each token becomes an ID from the vocabulary. “bank” is ID 2924.
- The embedding table has one row per vocabulary entry and one column per dimension: 30,522 × 384 learned numbers.
- Embedding “bank” means reading row 2924. No arithmetic, just a lookup. The same ID always reads the same row, whatever the sentence.
Tokens, IDs, table shape and row values measured with sentence-transformers 6.1.0 (all-MiniLM-L6-v2).
There is no arithmetic in this step. "Embed token 2924" means "read row 2924". A sequence of N token IDs becomes N vectors, one per token.
A real table
The small embedding model used throughout this lesson, sentence-transformers/all-MiniLM-L6-v2, has a BERT-style tokenizer. Inspecting it with sentence-transformers 6.1.0 shows:
vocabulary size: 30,522
embedding dimension: 384
token embedding table: 30,522 × 384 = 11,720,448 numbers
token ID of "bank": 2924
Every one of those 11.7 million numbers is a trained parameter. Before training, the table holds random values like the rest of the model's weights; training nudges each row until the vectors are useful for the task the model is trained on. A pretrained model ships with its table, which is why a model is tied to the tokenizer it was trained with: swap the tokenizer and the IDs point at the wrong rows.
Predict first