Mid-level interview questionAsked at Snap · interview write-up
Design text search over short videos when only about 20% of them have reliable text metadata.
Model answer
Don't search only the 20%: make the rest searchable. Embed frames and queries in a shared space with a CLIP-style model, and generate text from speech recognition, OCR and vision-language captions. Index both: BM25 over real and generated text, and an ANN index over visual embeddings, merged and reranked. Fine-tune the dual encoder on search logs of query and watched video. Report recall at k separately for videos with and without metadata.
Practise the rest
12 Multimodal questions, 492 in all, each with a model answer. Free.
More from Snap interview questions.
More Multimodal interview questions
- What has driven each generation of LLM improvements, how do multimodal LLMs work, and what role does normalisation play in LLMs?
- You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?
- How does a vision encoder feed images into a multimodal LLM, and where are the bottlenecks?
- Design the training of a multimodal embedding model: architecture, supervision and losses, evaluation metrics and deployment.
- Turn a multimodal model that captions videos into a production service under tight compute and GPU memory limits.
- Design a classifier that assigns genres to games using text, images and tabular metadata, from label quality to serving and monitoring.