Mid-level interview questionAsked at TikTok · interview write-up
Design the training of a multimodal embedding model: architecture, supervision and losses, evaluation metrics and deployment.
Model answer
Two encoders, image and text, project into one shared space. Train on pairs with a contrastive InfoNCE loss, as CLIP does: each image must outscore the batch with its own caption, so big batches and hard negatives matter.
Evaluate with retrieval recall at k both ways and zero-shot classification. To deploy, index item embeddings for ANN search and encode queries online. Version the index with the model: embeddings from two versions aren't comparable.
Practise the rest
12 Multimodal questions, 492 in all, each with a model answer. Free.
More from TikTok interview questions.
More Multimodal interview questions
- What has driven each generation of LLM improvements, how do multimodal LLMs work, and what role does normalisation play in LLMs?
- You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?
- How does a vision encoder feed images into a multimodal LLM, and where are the bottlenecks?
- Turn a multimodal model that captions videos into a production service under tight compute and GPU memory limits.
- Design a classifier that assigns genres to games using text, images and tabular metadata, from label quality to serving and monitoring.
- For an offline audio detection system, outline the ML inference and labeling pipeline: feature extraction, keyword spotting, thresholding, calibration and human labeling.