Senior interview questionAsked at Microsoft · interview write-up
An input embedding table is too large for one device. Compare sharding it by vocabulary with sharding it by hidden dimension.
Model answer
By vocabulary, each device owns a range of rows, looks up the tokens it owns, writes zeros for the rest, and an all-reduce sum assembles full vectors. It pairs with a tied output head sharded the same way, but popular tokens can make one shard hot.
Scroll sideways to see the whole diagram
MiniLM's tokenizer lowercases and splits “the river bank”, then adds two special tokens, [CLS] and [SEP].
Text version
- MiniLM's tokenizer lowercases and splits “the river bank”, then adds two special tokens, [CLS] and [SEP].
- Each token becomes an ID from the vocabulary. “bank” is ID 2924.
- The embedding table has one row per vocabulary entry and one column per dimension: 30,522 × 384 learned numbers.
- Embedding “bank” means reading row 2924. No arithmetic, just a lookup. The same ID always reads the same row, whatever the sentence.
Tokens, IDs, table shape and row values measured with sentence-transformers 6.1.0 (all-MiniLM-L6-v2).
By hidden dimension, every device holds all rows but a slice of the width. Lookups are local, then an all-gather concatenates the slices. Both move about batch × tokens × width numbers; choose by what the next layer wants.
Practise the rest
17 Embeddings questions, 492 in all, each with a model answer. Free.
More from Microsoft interview questions.
More Embeddings interview questions
- What is an embedding table, and what does it mean for two embeddings to be similar?
- People say 'embedding' for three different things. What are they, and why does the difference matter?
- Your vector database only supports dot-product search, but your embedding model's documentation says to compare vectors with cosine similarity. What do you do, and why does it work?
- You're building search over medicine leaflets with a small English embedding model, and you plan to reply 'no answer found' whenever the top cosine score is below 0.5. What risks do you raise before this ships?
- Your team switches to a newer embedding model with the same 384 dimensions and starts embedding incoming queries with it. Search quality collapses, but nothing errors. Explain what happened and how you'd run the migration.
- Embedding models return 1,536 or 3,072 numbers per text. What do those sizes mean?