Junior interview question
What is an embedding table, and what does it mean for two embeddings to be similar?
Model answer
It's a learned matrix with one row per vocabulary entry, and embedding a token just reads its row. all-MiniLM-L6-v2's table is 30,522 × 384, about half its parameters.
Scroll sideways to see the whole diagram
MiniLM's tokenizer lowercases and splits “the river bank”, then adds two special tokens, [CLS] and [SEP].
Text version
- MiniLM's tokenizer lowercases and splits “the river bank”, then adds two special tokens, [CLS] and [SEP].
- Each token becomes an ID from the vocabulary. “bank” is ID 2924.
- The embedding table has one row per vocabulary entry and one column per dimension: 30,522 × 384 learned numbers.
- Embedding “bank” means reading row 2924. No arithmetic, just a lookup. The same ID always reads the same row, whatever the sentence.
Tokens, IDs, table shape and row values measured with sentence-transformers 6.1.0 (all-MiniLM-L6-v2).
No single coordinate means anything; training shapes which vectors land near which. "Similar" means pointing the same way, usually measured with cosine similarity.
Practise the rest
17 Embeddings questions, 492 in all, each with a model answer. Free.
More Embeddings interview questions
- People say 'embedding' for three different things. What are they, and why does the difference matter?
- Your vector database only supports dot-product search, but your embedding model's documentation says to compare vectors with cosine similarity. What do you do, and why does it work?
- You're building search over medicine leaflets with a small English embedding model, and you plan to reply 'no answer found' whenever the top cosine score is below 0.5. What risks do you raise before this ships?
- Your team switches to a newer embedding model with the same 384 dimensions and starts embedding incoming queries with it. Search quality collapses, but nothing errors. Explain what happened and how you'd run the migration.
- Embedding models return 1,536 or 3,072 numbers per text. What do those sizes mean?
- An input embedding table is too large for one device. Compare sharding it by vocabulary with sharding it by hidden dimension.