Junior interview question
People say 'embedding' for three different things. What are they, and why does the difference matter?
Model answer
Only the last two change with context:
- Static token embedding: a table row, identical in every sentence: river "bank" and money "bank" score cosine exactly 1.
- Contextual token embedding: the transformer's output at one position, so it depends on context.
- Text embedding: one vector per text; MiniLM averages its token outputs, then normalizes.
Scroll sideways to see the whole diagram
The tokenizer turns the sentence into 9 token IDs, including the special tokens [CLS] and [SEP].
Text version
- The tokenizer turns the sentence into 9 token IDs, including the special tokens [CLS] and [SEP].
- Each ID reads its row of the embedding table: 9 static vectors. “bank” gets the same row here as in a sentence about money.
- Six transformer layers let every token's vector take in the others. The output is still 9 vectors, but now each depends on its context.
- Search needs one vector per text, so MiniLM averages the 9 token vectors (mean pooling) into one 384-number vector.
- Finally the vector is scaled to length 1, so cosine similarity and dot product give the same ranking.
Token count, module order (Transformer → mean Pooling → Normalize) and output length measured with sentence-transformers 6.1.0 (all-MiniLM-L6-v2).
Practise the rest
17 Embeddings questions, 492 in all, each with a model answer. Free.
More Embeddings interview questions
- What is an embedding table, and what does it mean for two embeddings to be similar?
- Your vector database only supports dot-product search, but your embedding model's documentation says to compare vectors with cosine similarity. What do you do, and why does it work?
- You're building search over medicine leaflets with a small English embedding model, and you plan to reply 'no answer found' whenever the top cosine score is below 0.5. What risks do you raise before this ships?
- Your team switches to a newer embedding model with the same 384 dimensions and starts embedding incoming queries with it. Search quality collapses, but nothing errors. Explain what happened and how you'd run the migration.
- Embedding models return 1,536 or 3,072 numbers per text. What do those sizes mean?
- An input embedding table is too large for one device. Compare sharding it by vocabulary with sharding it by hidden dimension.