ML system design interview questions

Q: Design a job scheduler for ML training and batch inference on a shared GPU cluster.

Jobs enter a queue with a priority and a team quota, with fair share across teams. Distributed training needs gang scheduling, all GPUs at once or none, and topology-aware placement on NVLink-connected GPUs in the same node or rack. Batch inference is the filler: preemptible, sharded into small tasks, backfilling idle GPUs. Preemption relies on checkpointing, and so does recovery: health checks spot failed GPUs, and jobs restart from their last checkpoint. Track utilisation, queue wait per priority and fairness, and give users a clear view of why a job is waiting.

Question 1

You need to copy 500 GB of weights to 1,000 GPU workers over shared 10 Gbps links. What's the lower bound on time, and what would you build?

Accepted Answer

python size = 500e9 bytes link = 10e9 / 8 1.25e9 bytes per second tfloor = size / link 400 s per receiver tnaive = 1000 tfloor 4e5 s from one source Each worker must receive 500 GB through its own 10 Gbps link, so 400 seconds is the floor; serving everyone from one source takes about 4.6 days. To approach the floor, every receiver also uploads: split weights into chunks and pipeline them down chains or trees, or swarm BitTorrent-style. Time becomes roughly 400 s plus a few chunk times per hop. Respect topology: one copy crosses each rack uplink, then fans out locally. Checksum chunks and resume on failure.

Question 2

Design an LLM coding assistant end to end: data, model and evaluation.

Accepted Answer

Data: permissively licensed code, deduplicated and filtered, with fill-in-the-middle training so the model completes inside a file, then instruction and repository-level data. Model: a small fast one for inline completion, a larger one for chat and edits. Serving: build context from the cursor's surroundings, open files and retrieved repo snippets, cache the prefix, and cancel stale requests as the user types.

Evaluation: execution-based benchmarks like HumanEval and repo-level tasks offline; acceptance rate and retained code online, through A/B tests.

Question 3

Design a job scheduler for ML training and batch inference on a shared GPU cluster.

Accepted Answer

Jobs enter a queue with a priority and a team quota, with fair share across teams. Distributed training needs gang scheduling, all GPUs at once or none, and topology-aware placement on NVLink-connected GPUs in the same node or rack.

Batch inference is the filler: preemptible, sharded into small tasks, backfilling idle GPUs. Preemption relies on checkpointing, and so does recovery: health checks spot failed GPUs, and jobs restart from their last checkpoint. Track utilisation, queue wait per priority and fairness, and give users a clear view of why a job is waiting.

Question 4

Training a retrieval model over a catalogue of millions of items, you use sampled softmax. Why does it need a correction, and how do you choose the classification loss?

Accepted Answer

A full softmax over millions of items is too costly, so each positive competes against sampled negatives. In-batch negatives appear in proportion to popularity, so popular items get over-penalised. The logQ correction subtracts each item's log sampling probability from its logit, which makes the sampled loss approximate the full softmax.

import torch
import torch.nn.functional as F

def in_batch_loss(u, items, log_q):
    # u, items: [B, d]; row i of items is user i's
    # positive; log_q: [B] log sampling probability
    logits = u @ items.T - log_q[None, :]
    labels = torch.arange(len(u), device=u.device)
    return F.cross_entropy(logits, labels)

Loss: softmax cross-entropy when exactly one item is right among many; per-item binary cross-entropy for multi-label targets or calibrated click probabilities.

Question 5

Design an inference API that converts PDFs into Markdown using several models in stages.

Accepted Answer

Split it into stages with different costs: render pages and detect layout, then recognise each region with the right model, such as text OCR, tables or formulas, then assemble Markdown in reading order. OCR only pages without a reliable text layer.

Make the API asynchronous: upload returns a job id, pages fan out on a queue, and a failed page retries alone. Batch GPU stages across jobs, keep a fast lane for small requests, and cache by file hash. Evaluate each stage, with edit distance to reference Markdown.

Question 6

How can LLMs be incorporated into recommendation systems, from metadata enrichment and cold start to generative retrieval and semantic reranking?

Accepted Answer

Use LLMs where they add knowledge ID models lack, off the hot path.

Offline enrichment: generate tags, summaries and embeddings from titles, descriptions and frames. That mostly helps cold start, since a new item has meaningful features before it has clicks. Generative retrieval: give items semantic IDs, codes from quantized content embeddings, and train a sequence model to generate the next item's ID, as in TIGER. Semantic reranking: a distilled LLM rescores the top few dozen candidates against the user's intent. Judge each by offline recall and NDCG, then A/B tests.

Question 7

In the recall stage of a recommender, design a model that chooses how many candidates K to retrieve per request given latency and compute cost.

Accepted Answer

Predict the marginal value of more candidates per request. Label offline by replaying traffic at several K values and recording where the final ranker's top results stop changing. Features: user activity, history length, request context and current load. A small model, say gradient-boosted trees, picks K from a few buckets to maximise value minus latency and compute cost, with a safe floor. A/B test against fixed K, watching p99 latency.

Question 8

Design a ride-hailing ETA system, covering labels, features, model choice, low-latency serving, delayed ground truth and drift.

Accepted Answer

Label: actual trip time. Start from a routing engine's ETA over the road graph, then learn a residual correction from time of day, live traffic, region, weather and trip type, as DeeprETA does. Serve a compact model with cached features. Ground truth lands when the trip ends, so track error by segment as labels arrive, watch feature drift meanwhile, and retrain regularly. Pick the loss by business cost, asymmetric if lateness hurts more.

Question 9

How would you build ranking for a food-delivery marketplace, from training labels and objectives to features, architecture and evaluation?

Accepted Answer

Two stages: retrieve a few hundred eligible stores, filtered by delivery radius and opening hours, then rank them.

Labels come from the funnel: click, add to cart, order, with orders the main target. A multi-task model predicts each step, and the final score blends them with business terms like delivery time and fees. Features: user history and cuisine affinity, store quality, ETA, price, time and location. Labels carry position bias, so log position and correct for it. Evaluate offline with NDCG against orders, then A/B test orders per session, gross bookings and retention.

Question 10

Design a real-time card fraud detector with a 0.2% fraud rate, 14-day label delay, a 50 ms p95 latency budget and an explicit cost matrix for false positives and negatives.

Accepted Answer

Score every authorisation inside the 50 ms budget: precompute velocity features, like recent spend per card, in a streaming feature store, and serve a gradient-boosted model in-process. Labels lag 14 days, so train only on transactions old enough to be labelled, and monitor with faster signals like customer disputes. With 0.2% fraud, keep probabilities calibrated and set the threshold from the cost matrix: decline when the expected fraud loss exceeds the expected cost of blocking a good customer. Add rules, a review queue for the grey zone, and shadow-test new models.

Question 11

You have trained a fraud model: how would you deploy it with versioning and safe rollbacks, monitor it after launch, and keep inference latency low?

Accepted Answer

Ship model, feature code and threshold as one versioned artifact in a registry. Run it in shadow mode, then canary it on a slice of traffic, with one-step rollback to the previous version.

Monitor latency, score distribution and feature drift, and precision and recall once labels arrive. For low latency: precomputed features, a compact model served in-process or on an optimised runtime, and a rules fallback if the model times out.

Question 12

Design a low-latency store recommender for a delivery app's homepage, covering retrieval, pre-ranking, ranking, geospatial caching and feature serving.

Accepted Answer

A cascade, cheap to expensive. Retrieval: stores that deliver to the user's location, from a geohash index, merged with two-tower candidates and popular-nearby lists. Pre-ranking trims them with a light model. Ranking runs a richer multi-task model over user, store and context features. Cache store lists and slow-changing features per geohash cell; ETA and store busyness come from a low-latency feature store. Give each stage a latency budget with timeouts, and fall back to the cached popular list.

Question 13

Target a free-delivery promotion at 20% of customers to maximize incremental orders per dollar: define the uplift estimand, the experiment to collect data and how you evaluate uplift models offline.

Accepted Answer

The estimand is the conditional average treatment effect: the extra orders free delivery causes for a customer, divided by its cost. Collect data with a randomised experiment, treating a random slice, so treatment is independent of features. Fit a T-learner, X-learner or uplift trees. You never see one customer's effect, so evaluate offline with Qini or uplift curves: rank holdout customers by predicted uplift and compare treated against control order rates in the top 20%, per dollar spent. Confirm with a live test of the targeted policy against random targeting.

Question 14

Design the architecture of a real-time game recommender for millions of games and tens of millions of users, with request flow and latency SLOs.

Accepted Answer

The client calls a recommendation service, which fetches user features and recent sessions, then retrieves candidates from several sources: a two-tower ANN index over millions of games, trending games, and friends' games. It ranks a few hundred with a multi-task model and applies filters and diversity before returning.

Set an end-to-end latency SLO and split it by stage, with timeouts and a cached fallback list. Stream play events into the feature store so recent sessions count within minutes, retrain daily, and refresh the item index as games launch.

Question 15

Design personalized discovery recommendations for an AI answer engine, from event logging and candidate retrieval to ranking and feedback loops.

Accepted Answer

Log impressions, clicks, dwell, follow-up questions and shares, with position, so you know what was shown and ignored. Retrieve candidates from embeddings of the user's past queries, trending stories and editorial picks, and rank with a model predicting engaged reads, not clicks alone. Add freshness and diversity rules.

Guard against feedback loops: reserve a little traffic for exploration and correct for position. Evaluate offline on time-split logs, then A/B test retention.

Question 16

Design a real-time system that picks and orders a personalized 10-track playlist from 10,000 candidates, with sequence-aware reranking and diversity constraints.

Accepted Answer

Score all 10,000 candidates with a fast pointwise model of user, context and track, keep the top hundred or so. Then build the list slot by slot: a sequence-aware model scores the remaining tracks given the ones already chosen, for flow in tempo, energy and genre.

Apply diversity constraints greedily, MMR-style, such as at most two tracks per artist. Evaluate offline on skips and completions, then A/B test listening time and skip rate.

Question 17

Design a recommendation system for augmented-reality lenses in a camera app, including cold start for new lenses and new users.

Accepted Answer

Two stages. Candidate generation from several sources: lenses near the user's history, trending lenses, lenses friends used, camera context. Then a ranker predicting try, capture and send. New lenses: content features like category, creator and a visual embedding, plus a small exploration slot. New users: regional popularity, adapting within the first sessions. Evaluate offline on a time split, then A/B test on sends, not just tries.

Question 18

Design a real-time system that shows the top-K points of interest in a user's map viewport as they pan and zoom, within a 150 ms p95 budget.

Accepted Answer

Split it into spatial retrieval and ranking, both cheap. Index POIs in a spatial grid such as geohash or S2 cells, and precompute the top candidates per cell and zoom level offline from popularity and quality, so a viewport query just unions a few cells' lists. Online, a light model reranks a few hundred candidates with user and time features. To hold 150 ms p95: cache cell lists at the edge and on the client, prefetch neighbouring cells while the user pans, and cancel stale requests. If the ranker runs late, serve the precomputed order.

Question 19

Design and build a candidate search system that ranks people from noisy profiles and free-text queries while enforcing hard criteria.

Accepted Answer

Separate hard filters from soft ranking. Parse profiles and the query into structured fields, like location, experience and skills, with an LLM extractor plus rules, and apply hard criteria as filters so no ranking can break them. Retrieve with hybrid search over profile text, then rank on skill match, seniority and recency. Show why each person matched. Evaluate on recruiter-labelled queries with NDCG, and count hard-criteria violations, which must be zero.

Question 20

What role should LLMs play in fraud detection compared with traditional supervised models, considering data types, accuracy, latency, cost and interpretability?

Accepted Answer

LLMs aren't the core scorer. Fraud signals are mostly tabular and behavioural: amounts, velocities, device and graph features. Gradient-boosted trees or deep models trained on labels do better there, score in milliseconds at tiny cost, and give stable feature attributions for explanations.

LLMs earn a place around the model: turning unstructured text such as dispute notes, merchant descriptions and chats into features, summarising cases for investigators, drafting reason codes, and drafting rules. Keep them off the authorization path, validate their features like any other, and guard against prompt injection in user-written text.

Question 21

Design a real-time account-takeover model inside payment authorization, handling delayed noisy labels, feature freshness and a tight latency budget.

Accepted Answer

Score at login and payment within tens of milliseconds. Features: device fingerprint, session behaviour, recent credential or address changes, and velocities, served from a streaming feature store and logged at decision time so training sees exactly what serving saw.

Labels arrive late and noisy, from chargebacks and customer reports, so train on matured windows plus faster proxies like failed step-up challenges. Rather than only block, pick actions by risk: allow, step-up authentication or hold, with thresholds set on expected loss, and a rules fallback on timeout.

Question 22

Set per-user spending limits with reinforcement learning: define the state, actions, transitions and reward, and explain how you'd train and evaluate the policy safely.

Accepted Answer

State: the user's risk score, history, balance, recent spend and current limit. Actions: a few discrete moves, such as keep, raise or lower by a step. Transitions come from user behaviour. Reward: margin on enabled spend minus expected fraud and credit losses, which arrive late, minus a penalty for declining good spend. Train offline from logged decisions with propensities, using a conservative method, and estimate the policy's value with off-policy estimators such as doubly robust before any live test. Then roll out to a small cohort, inside hard caps set by people.

Question 23

Build a product ranking model for an e-commerce marketplace: what data you collect, how you define labels, which features and model you use, and how you evaluate.

Accepted Answer

Log impressions with position, plus clicks, add-to-carts and purchases, with query and context. Labels: graded relevance, purchase above add-to-cart above click above skip, corrected for position bias, since top slots get clicked regardless. Features: query-product text match, price, ratings, popularity, freshness and user affinity. Model: gradient-boosted trees with a LambdaMART objective, then a neural ranker if it pays. Evaluate with NDCG on later days, then A/B test on conversion and revenue per search.

Question 24

Design an end-to-end fraud detection model, covering label definition, severe class imbalance, label delay, metrics and the thresholds for action.

Accepted Answer

Labels: chargebacks and confirmed fraud are positives, but they arrive weeks later, so train only on orders whose labels have matured. Imbalance: keep every positive, downsample negatives, then re-weight or recalibrate. Features: payment, device, address mismatches and velocities, in gradient-boosted trees. Metrics: precision-recall AUC, not accuracy, and recall at your review capacity. Thresholds come from costs: approve, review or block, set by expected fraud loss against lost good orders.

Question 25

Design an item category prediction system for a large catalogue, handling hierarchical labels, multimodal features and label quality.

Accepted Answer

Predict a path in the taxonomy, like Home, Kitchen, Cookware. Encode the title, description and image and combine them. Predict level by level, or the leaf with a loss that respects the tree so mistakes stay nearby. Seller labels are noisy, so clean them with agreement checks, and send low-confidence items to review.

Question 26

Sales has limited outreach capacity. Design a lead-scoring model that ranks incoming leads, from labels and features to evaluation against that limit.

Accepted Answer

A lead is positive if it converted within a set window. Features: company data, source and engagement like visits and replies. Train a calibrated classifier, such as logistic regression or boosted trees, and rank by probability. If sales can call 100 a week, measure precision at 100 against the current process.

Question 27

Design a two-stage retrieval and ranking system, built on user embeddings, that surfaces the rental listings a user is most likely to book.

Accepted Answer

Candidate retrieval: a two-tower model embeds the user, from search context and booking and click history, and each listing, from attributes, location and engagement. Train it on bookings as positives with in-batch negatives; an ANN index returns a few thousand listings, filtered to the searched market and available dates. Ranking: a richer model scores those candidates with price, availability and user-listing cross features, predicting booking probability. Cold-start listings rely on content features. Judge offline with NDCG on held-out bookings, then run an A/B test on bookings.

Question 28

There is no text query, only structured intent such as location, dates and guests. Design the ranker for home search results, including bias from past rankings, cold start and online evaluation.

Accepted Answer

Intent is structured: location, dates, guests, filters, plus user history. Rank by predicted booking probability from listing, user and context features.

Logged data reflects past rankings, since people click what was shown first, so correct for position bias: train with position as a feature and fix it to a constant at serving, or weight examples by inverse propensity estimated from randomised slots. New listings get content-based priors and a little exploration traffic. Evaluate offline with NDCG on bookings, then A/B test bookings and cancellations, watching new-host exposure.

Question 29

Design an ML system that labels listings as family-friendly so guests can filter on it, from labeling strategy to features, evaluation and launch.

Accepted Answer

Define the label first, in a written policy: amenities like cribs and high chairs, hazards like unfenced pools. Labels come from host attributes, guest reviews mentioning children, and trained annotators on a stratified sample, with agreement measured.

Features: amenities, listing text and photos through a multimodal model, review signals. A wrong yes hurts a family more than a missed listing, so tune for high precision and send uncertain listings to review. Evaluate per region, launch as an A/B test on filter use, bookings and complaints, and let hosts dispute labels.

Question 30

Design a recommendation system for issues in an issue tracker, covering labels, candidate generation, ranking, real-time serving and permission-aware access.

Accepted Answer

Pick the label first: an issue the user then opens, comments on or picks up. Candidate generation pulls from the user's projects, issues linked to recent work, similar issues by embedding, and teammates' activity. A ranker scores them on recency, priority, relation to the user and text similarity. Filter by permission before ranking, and refresh cached candidates on events. Evaluate with recall and NDCG on held-out interactions, then A/B test.

Question 31

Design a system that routes incoming bug reports to the right team, using text and log features, labels from past assignments and a human fallback.

Accepted Answer

Labels come from historical assignments, but tickets get reassigned, so use the team that finally resolved each one and drop teams that no longer exist. Features: title and description embeddings, stack frames and error signatures from logs, component and reporter. Route automatically only when confidence clears a threshold; otherwise a triager picks from the top three, and their choice becomes a label. Track per-team accuracy and reassignment rate.

Question 32

Design a real-time home feed ranker that blends long-term personalization with streaming engagement signals, at low latency.

Accepted Answer

A funnel with a latency budget per stage. Retrieval pulls a few thousand candidates from several sources: ANN over item embeddings using a daily long-term user embedding, plus candidates seeded by the user's last few actions. A light pre-ranker trims them; a multi-task ranker predicts save, click and hide, combined into one score; a blender enforces diversity and freshness. Streaming events update a recent-action sequence in the online feature store within seconds; the ranker reads it alongside long-term features. Log features as served, so training matches serving. Judge by A/B tests.

Question 33

You own feed ads ranking. Design an ML system to raise ad click-through rate, from the metrics you'd pick to the model architecture and serving.

Accepted Answer

CTR alone is gameable: clickbait raises it. The auction ranks by bid times predicted CTR, so pCTR must be calibrated, not just well ordered. Offline: log loss, normalized entropy, AUC and calibration by slice. Online: CTR, revenue, advertiser conversions, and hide rate.

Architecture: ad retrieval by targeting and embeddings, then a deep ranker with feature crosses over sparse ID embeddings for user, ad, advertiser and context, with multi-task heads for click and conversion. Correct for position bias. Retrain continuously as ads churn, and recalibrate after each model change.

Question 34

Design a ranker for ads launched in the last seven days that handles cold start with safe exploration, then hands off smoothly to the mature model.

Accepted Answer

New ads have no ID history, so predict from what generalizes: creative embeddings, advertiser, category and targeting, with priors from similar ads. Explore with a capped traffic budget and Thompson sampling behind a quality floor.

Hand off by blending both models' scores, shifting weight to the mature model as impressions accumulate. Both must stay calibrated, since the auction uses the score directly. Measure later CTR and the revenue cost of exploring.

Question 35

Design a short-video recommender end to end, starting with which labels to optimize: watch time, completion, shares, hides and so on.

Accepted Answer

No single label works: raw watch time favours long videos, completion favours short ones. So predict several labels with a multi-task ranker: watch time normalized by duration, completion, like, share, follow, and hides or fast skips as negatives, then combine them with weights tuned against long-term retention in A/B tests.

The pipeline: two-tower retrieval over video embeddings, the ranker, then re-ranking for diversity and freshness. Model position bias separately so the ranker learns preference.

Question 36

Design a personalized feed ranker and a natural-language-to-HTML generation system. Cover objectives, cold start, exploration and evaluation for each.

Accepted Answer

Feed: optimize meaningful engagement, like opens and remixes, checked against retention. Retrieve, then rank; cold-start users get content embeddings and popular-by-segment items, plus a small exploration slot run as a bandit. Evaluate offline with NDCG, then A/B. HTML: an LLM with design-system components in context and constrained output. Check that the output parses and renders, then compare rendered screenshots against references with a calibrated judge. Online, track how often users keep the result.

Question 37

Design a system that predicts how likely a user is to comment on a post, focusing on label definition, features and model choice.

Accepted Answer

Label: did the user comment within a fixed window after a real impression, one actually viewed, so unseen posts aren't false negatives.

Features, taken as of impression time: user history and community affinity; post age, community, text embedding and comment velocity; author affinity; context. Start with gradient-boosted trees, then a multi-task network that learns comments alongside clicks and votes, since comments are sparse. Evaluate log loss, calibration and AUC, then comments per user online.

Question 38

Design a feature store that serves both offline training and low-latency online inference, with point-in-time correctness and offline-online consistency.

Accepted Answer

One feature definition, in versioned code, compiled to both batch and streaming jobs. They write an append-only, timestamped offline store and a key-value online store holding the latest value per entity. Training uses an as-of join: each label gets the latest feature value at or before its event time, never after. For consistency, log served features and compare them with offline recomputation to catch skew. CI tests transforms, schemas and backfills. In production, replicate the online store, give every feature a default and a timeout, and alert on freshness, null rates and drift.

Question 39

Design a video recommendation system for the home feed and up-next, focusing on infrastructure, logging and observability.

Accepted Answer

One stack, two contexts: home feed has no seed; up-next adds the current video. Candidates come from ANN over video embeddings, co-watch, subscribed communities and trending, then a GPU ranker reads the online feature store.

Logging is the backbone: every impression records request ID, model version, candidates, scores, position and features as served. Client events like watch seconds and skips join by request ID into skew-free training data.

Observability: per-stage latency SLOs, candidate counts per source, empty results, feature freshness, score and calibration drift, and per-surface business metrics. Ship via canary with automatic rollback.

Question 40

Frame named entity recognition for short e-commerce text like listing titles: which algorithm, what data processing, and which metrics?

Accepted Answer

It's token classification with BIO tags for brand, model, size, colour. Titles are short and noisy, so a fine-tuned BERT-style encoder beats a CRF baseline. Normalize text, write labelling guidelines, and bootstrap labels from structured item attributes. Measure entity-level precision, recall and F1 per type, counting only exact span and type matches.