Senior interview questionAsked at Apple · interview write-up
How does a vision encoder feed images into a multimodal LLM, and where are the bottlenecks?
Model answer
A vision encoder, usually a ViT trained contrastively like CLIP, cuts the image into patches and embeds each one. A projector, an MLP or a cross-attention resampler, maps those embeddings into the LLM's space, and the LLM reads them as tokens.
The main bottleneck is visual token count: a 336-pixel image with 14-pixel patches is 576 tokens, and high-resolution tiling multiplies that, inflating prefill and the KV cache. Low fixed resolution loses small text, and contrastive encoders capture weak spatial detail. Fixes: token pooling, dynamic tiling, and fine-tuning the encoder.
Practise the rest
12 Multimodal questions, 492 in all, each with a model answer. Free.
More from Apple interview questions.
More Multimodal interview questions
- What has driven each generation of LLM improvements, how do multimodal LLMs work, and what role does normalisation play in LLMs?
- You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?
- Design the training of a multimodal embedding model: architecture, supervision and losses, evaluation metrics and deployment.
- Turn a multimodal model that captions videos into a production service under tight compute and GPU memory limits.
- Design a classifier that assigns genres to games using text, images and tabular metadata, from label quality to serving and monitoring.
- For an offline audio detection system, outline the ML inference and labeling pipeline: feature extraction, keyword spotting, thresholding, calibration and human labeling.