Senior interview questionAsked at Microsoft · interview write-up
Your assistant's semantic search over user content is slow and expensive. How do you optimise it: embedding choice, ANN indexing and sharding?
Model answer
Embeddings: shortlist models on public retrieval benchmarks, then pick on your own queries; smaller dimensions or int8 vectors cut memory and latency.
Scroll sideways to see the whole diagram
The tokenizer turns the sentence into 9 token IDs, including the special tokens [CLS] and [SEP].
Text version
- The tokenizer turns the sentence into 9 token IDs, including the special tokens [CLS] and [SEP].
- Each ID reads its row of the embedding table: 9 static vectors. “bank” gets the same row here as in a sentence about money.
- Six transformer layers let every token's vector take in the others. The output is still 9 vectors, but now each depends on its context.
- Search needs one vector per text, so MiniLM averages the 9 token vectors (mean pooling) into one 384-number vector.
- Finally the vector is scaled to length 1, so cosine similarity and dot product give the same ranking.
Token count, module order (Transformer → mean Pooling → Normalize) and output length measured with sentence-transformers 6.1.0 (all-MiniLM-L6-v2).
Index: HNSW gives high recall at low latency but holds full vectors in memory; IVF with product quantisation compresses billions of vectors. Tune the recall-latency trade with ef or nprobe, then rerank a shortlist exactly.
Sharding: partition by user or tenant, since an assistant searches one person's content, which also enforces isolation; otherwise scatter-gather across shards and merge top-k. Add replicas for throughput.
Practise the rest
17 Embeddings questions, 492 in all, each with a model answer. Free.
More from Microsoft interview questions.
More Embeddings interview questions
- What is an embedding table, and what does it mean for two embeddings to be similar?
- People say 'embedding' for three different things. What are they, and why does the difference matter?
- Your vector database only supports dot-product search, but your embedding model's documentation says to compare vectors with cosine similarity. What do you do, and why does it work?
- You're building search over medicine leaflets with a small English embedding model, and you plan to reply 'no answer found' whenever the top cosine score is below 0.5. What risks do you raise before this ships?
- Your team switches to a newer embedding model with the same 384 dimensions and starts embedding incoming queries with it. Search quality collapses, but nothing errors. Explain what happened and how you'd run the migration.
- Embedding models return 1,536 or 3,072 numbers per text. What do those sizes mean?