Senior interview questionAsked at Pinterest · interview write-up
You own feed ads ranking. Design an ML system to raise ad click-through rate, from the metrics you'd pick to the model architecture and serving.
Model answer
CTR alone is gameable: clickbait raises it. The auction ranks by bid times predicted CTR, so pCTR must be calibrated, not just well ordered. Offline: log loss, normalized entropy, AUC and calibration by slice. Online: CTR, revenue, advertiser conversions, and hide rate.
Architecture: ad retrieval by targeting and embeddings, then a deep ranker with feature crosses over sparse ID embeddings for user, ad, advertiser and context, with multi-task heads for click and conversion. Correct for position bias. Retrain continuously as ads churn, and recalibrate after each model change.
Practise the rest
40 ML system design questions, 492 in all, each with a model answer. Free.
More from Pinterest interview questions.
More ML system design interview questions
- You need to copy 500 GB of weights to 1,000 GPU workers over shared 10 Gbps links. What's the lower bound on time, and what would you build?
- Design an LLM coding assistant end to end: data, model and evaluation.
- Design a job scheduler for ML training and batch inference on a shared GPU cluster.
- Training a retrieval model over a catalogue of millions of items, you use sampled softmax. Why does it need a correction, and how do you choose the classification loss?
- Design an inference API that converts PDFs into Markdown using several models in stages.
- How can LLMs be incorporated into recommendation systems, from metadata enrichment and cold start to generative retrieval and semantic reranking?