Senior interview questionAsked at Nuro · interview write-up
Design a production ML inference pipeline covering request routing to models, artifact versioning and deployment, and online feature retrieval.
Model answer
A router consults a model registry that maps each model name to versions, endpoints and traffic splits, enabling canary, shadow and A/B.
Artifacts are immutable and versioned together: weights, preprocessing and feature schema, with lineage to data and code. CI checks offline metrics, schema compatibility and latency before promotion; canaries roll back automatically on SLO breaches.
Features come from an online store by entity key, with timeouts and defaults, using the same transforms as training; log what was served. Batch dynamically and autoscale on queue depth.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from Nuro interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?