Senior interview questionAsked at Netflix · interview write-up
Design a job scheduler for ML training and batch inference on a shared GPU cluster.
Model answer
Jobs enter a queue with a priority and a team quota, with fair share across teams. Distributed training needs gang scheduling, all GPUs at once or none, and topology-aware placement on NVLink-connected GPUs in the same node or rack.
Batch inference is the filler: preemptible, sharded into small tasks, backfilling idle GPUs. Preemption relies on checkpointing, and so does recovery: health checks spot failed GPUs, and jobs restart from their last checkpoint. Track utilisation, queue wait per priority and fairness, and give users a clear view of why a job is waiting.
Practise the rest
40 ML system design questions, 492 in all, each with a model answer. Free.
More from Netflix interview questions.
More ML system design interview questions
- You need to copy 500 GB of weights to 1,000 GPU workers over shared 10 Gbps links. What's the lower bound on time, and what would you build?
- Design an LLM coding assistant end to end: data, model and evaluation.
- Training a retrieval model over a catalogue of millions of items, you use sampled softmax. Why does it need a correction, and how do you choose the classification loss?
- Design an inference API that converts PDFs into Markdown using several models in stages.
- How can LLMs be incorporated into recommendation systems, from metadata enrichment and cold start to generative retrieval and semantic reranking?
- In the recall stage of a recommender, design a model that chooses how many candidates K to retrieve per request given latency and compute cost.