Service level objective (SLO)

Also called service level objective, SLO, SLOs

A target a service promises to meet, such as 'first token within 100 ms and at most 10 ms per token after that'.

Learn it in the lessonGoodput: counting only what meets the targetInference and Model Optimization · Measuring Inference: Latency, Throughput and Goodput

Interview questions about Service level objective (SLO)

Design the architecture of a real-time game recommender for millions of games and tens of millions of users, with request flow and latency SLOs.
The client calls a recommendation service, which fetches user features and recent sessions, then retrieves candidates from several sources: a two-tower ANN…
Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each…
Design the routing and scheduling layer in front of an inference fleet that serves many tenants.
Three jobs. Admission: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch.
Design a production ML inference pipeline covering request routing to models, artifact versioning and deployment, and online feature retrieval.
A router consults a model registry that maps each model name to versions, endpoints and traffic splits, enabling canary, shadow and A/B.
(Illustrative numbers.) Model A costs $0.004 per request and is correct 50% of the time; model B costs $0.006 and is correct 95% of the time. Which is cheaper, and what should you track?
B. Track cost per useful answer: spend divided by answers that were on time and correct.
Design optimised inference pipelines, and tell me how you would benchmark them.
Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt.
Design a video recommendation system for the home feed and up-next, focusing on infrastructure, logging and observability.
One stack, two contexts: home feed has no seed; up-next adds the current video.

Taught alongside

Glossary

Service level objective (SLO)

Also called service level objective, SLO, SLOs

A target a service promises to meet, such as 'first token within 100 ms and at most 10 ms per token after that'.

Learn it in the lessonGoodput: counting only what meets the targetInference and Model Optimization · Measuring Inference: Latency, Throughput and Goodput

Interview questions about Service level objective (SLO)

Design the architecture of a real-time game recommender for millions of games and tens of millions of users, with request flow and latency SLOs.
The client calls a recommendation service, which fetches user features and recent sessions, then retrieves candidates from several sources: a two-tower ANN…
Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each…
Design the routing and scheduling layer in front of an inference fleet that serves many tenants.
Three jobs. Admission: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch.
Design a production ML inference pipeline covering request routing to models, artifact versioning and deployment, and online feature retrieval.
A router consults a model registry that maps each model name to versions, endpoints and traffic splits, enabling canary, shadow and A/B.
(Illustrative numbers.) Model A costs $0.004 per request and is correct 50% of the time; model B costs $0.006 and is correct 95% of the time. Which is cheaper, and what should you track?
B. Track cost per useful answer: spend divided by answers that were on time and correct.
Design optimised inference pipelines, and tell me how you would benchmark them.
Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt.
Design a video recommendation system for the home feed and up-next, focusing on infrastructure, logging and observability.
One stack, two contexts: home feed has no seed; up-next adds the current video.