Embeddings interview questions

It's a learned matrix with one row per vocabulary entry, and embedding a token just reads its row. all-MiniLM-L6-v2's table is 30,522 × 384, about half its parameters.

No single coordinate means anything; training shapes which vectors land near which. "Similar" means pointing the same way, usually measured with cosine similarity.

Only the last two change with context:

• Static token embedding: a table row, identical in every sentence: river "bank" and money "bank" score cosine exactly 1.
• Contextual token embedding: the transformer's output at one position, so it depends on context.
• Text embedding: one vector per text; MiniLM averages its token outputs, then normalizes.

Normalize every vector, documents and queries alike, to length 1. Cosine is the dot product divided by both lengths, so on unit vectors the two are the same number.

Without that, the dot product also rewards length, so a long document can outrank a closer short one. Verify rather than assume: for the lesson's sentences, MiniLM's vectors were 4.86 to 6.46 long until its final normalize step.

Three risks. First, similarity tracks topic, not truth: in our MiniLM data, "This medicine is safe for children." and its negation scored 0.936, far above the best real paraphrase pair at 0.645. For medicine, negation needs explicit tests.

Second, 0.5 means nothing universal. Scores aren't calibrated and there's always a nearest neighbour, so pick the threshold from labelled real queries and recheck it when the model changes.

Third, MiniLM truncates after 256 word pieces according to its model card, so chunk long leaflets.

Each model arranges its own space, so matching dimensions only means the dot product can be computed. New-model queries against old-model documents give effectively random scores, with no error.

Migrate by re-embedding the whole collection, cutting queries and documents over together, and re-checking every threshold, since thresholds are model-specific. Even a preprocessing change invalidates stored vectors. Budget for it: generating embeddings costs money, and storage grows with dimension. At 4-byte floats, a million documents is about 1.5 GB at 384 dimensions against 12.3 GB at 3,072.

That's the vector's dimension: each text becomes a point with that many coordinates. More dimensions can hold finer distinctions, but every vector costs more memory and search time. The numbers mean nothing alone: vectors compare only with others from the same model.

By vocabulary, each device owns a range of rows, looks up the tokens it owns, writes zeros for the rest, and an all-reduce sum assembles full vectors. It pairs with a tied output head sharded the same way, but popular tokens can make one shard hot.

By hidden dimension, every device holds all rows but a slice of the width. Lookups are local, then an all-gather concatenates the slices. Both move about batch × tokens × width numbers; choose by what the next layer wants.

Embeddings: shortlist models on public retrieval benchmarks, then pick on your own queries; smaller dimensions or int8 vectors cut memory and latency.

Index: HNSW gives high recall at low latency but holds full vectors in memory; IVF with product quantisation compresses billions of vectors. Tune the recall-latency trade with ef or nprobe, then rerank a shortlist exactly.

Sharding: partition by user or tenant, since an assistant searches one person's content, which also enforces isolation; otherwise scatter-gather across shards and merge top-k. Add replicas for throughput.

Two endpoints, one encoder: classification is a light head on the embedding. Interactive calls are synchronous, with dynamic batching to a latency target.

Batch is an async job: submit a file, get a job id, poll or take a webhook, read results from storage, trading latency for cost. Every response names the model version, because vectors from different versions aren't comparable.

Build a collaboration graph: edges from shared documents, meetings, messages, code reviews and the org chart, weighted by frequency and recency. Learn node embeddings, with node2vec or a graph neural network, combine them with text embeddings of each person's work, and use cosine distance.

Validate by predicting held-out future collaborations, then by people-search clicks. Respect permissions: only use signals the viewer could see.

Yes: a two-tower model with user and item embeddings, trained with in-batch negatives. A positive should mean value: a click with real dwell time or a save, never a bare impression.

Retrain on a time-based split, computing features only from data before each label's timestamp, so nothing leaks from the future. Exposure bias: the logs only hold what the old model chose to show, so keep a small exploration slice of randomised items, correct for popularity in sampled negatives, and weight by inverse propensity. Validate on the following day, then A/B test.

In high dimensions data gets sparse and distances concentrate, so models need far more data. PCA finds the linear projection that keeps the most variance: fast, deterministic, and easy to size from explained variance. An autoencoder learns a nonlinear encoder and decoder by minimising reconstruction error. A linear autoencoder with squared loss spans the same subspace as PCA, so it only wins on nonlinear structure with enough data. Judge them on the downstream task, not reconstruction: train the same model on each representation, sweep the dimension, and compare held-out accuracy and stability across seeds.

import numpy as np

def top_k_cosine(q, docs, k=5, eps=1e-12):
    n = np.linalg.norm(docs, axis=1, keepdims=True)
    d = docs / np.maximum(n, eps)
    qn = q / max(np.linalg.norm(q), eps)
    scores = d @ qn
    idx = np.argsort(-scores)[:k]
    return idx, scores[idx]

Cosine is the dot product of unit-length vectors, so normalise once at indexing and any dot-product index ranks by cosine. Raw dot product also rewards vector length. Follow how the embedding model was trained: cosine for most sentence-embedding models, dot product when the model puts meaningful signal, such as popularity, in the norm. Guard against zero vectors.

A two-tower model. The item tower embeds image and text content, so new items get vectors immediately. The user tower runs a transformer over recent actions plus pooled summaries of older history, since 160,000 events won't fit attention.

Build each example as of its timestamp: only earlier events, split train and test by time. For freshness, refresh user embeddings daily and add recent actions at request time. New users fall back to onboarding interests.

Join each impression to the profile as of that timestamp, label clicks positive and skips negative, then split by time: train on the past, validate on the next window, and keep anything fitted, like vocabularies or normalizers, to the training slice.

Model: a two-tower text encoder for member and item, trained contrastively with in-batch negatives plus skips as hard negatives. Evaluate recall at k against a popularity baseline.

For reproducibility, pin seeds, model versions and a data-snapshot hash in one config, ship preprocessing with the model, and test that inference reproduces the training embeddings.

Uploads land in object storage and enqueue a job. Workers per modality: documents are parsed and chunked by structure; images are embedded directly; videos are sampled at intervals and scene changes, with the audio transcribed and embedded as text. Batch on GPUs.

Idempotency: key each output by content hash, model version and chunking config, and upsert, so retries never duplicate. Failures go to a dead-letter queue.

Store vectors with tenant and offsets. Tenant isolation is enforced server-side: per-tenant namespaces, per-tenant keys, and deletes that propagate to vectors. Model upgrades re-embed into a new index.

Text arrives on a log. Stream workers tokenize with the tokenizer pinned to the encoder version, embed in GPU micro-batches, and store vectors with ID, timestamp and version.

Train a light head, logistic regression or an MLP, on frozen embeddings, joining labels by time; retrain periodically and promote only after validating on the newest window.

Serving latency is mostly the encoder: use a small distilled one, batch, and cache repeated texts. Keep training and serving on one embedding version, and backfill when it changes.