Dot product
Also called dot products
Multiply two vectors element by element and add the results. Grows with both their alignment and their lengths.
Interview questions about Dot product
- In a notebook, implement embedding retrieval with cosine similarity. How does cosine relate to the dot product, and when would you choose each?
- Your vector database only supports dot-product search, but your embedding model's documentation says to compare vectors with cosine similarity. What do you do, and why does it work?
Normalize every vector, documents and queries alike, to length 1.
- Your team switches to a newer embedding model with the same 384 dimensions and starts embedding incoming queries with it. Search quality collapses, but nothing errors. Explain what happened and how you'd run the migration.
Each model arranges its own space, so matching dimensions only means the dot product can be computed.
- Walk me through scaled dot-product attention. What are queries, keys and values, and why divide by the square root of d?
Score each token's query against every visible key, divide by √d, softmax, and weight the values.
- Using only plain Python loops, implement a numerically stable sigmoid, a stable softmax and single-head scaled dot-product attention.
- Explain FlashAttention, the KV cache and RoPE, and the memory and compute trade-offs each addresses.
FlashAttention computes exact attention in tiles kept in on-chip SRAM, with an online softmax, so the full score matrix never goes to HBM.
- What is attention, exactly how are the queries, keys and values computed, and how does a KV cache speed up decoding?
Attention builds each token's output as a weighted mix of other tokens.
- Build a minimal embedding-based RAG pipeline: chunk, index, retrieve and assemble the prompt. How do you evaluate retrieval, resist injected instructions and abstain when unsure?