Senior interview questionAsked at ByteDance · interview write-up
You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?
Model answer
Profile first: preprocessing, vision encoder, prefill and decode each take their share. Then cut visual tokens, the big lever, by lowering resolution, pooling or merging tokens; quantise weights; cache image embeddings; and run encoder and LLM as separate stages with continuous batching, scaled independently.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
For video, do the heavy work offline: split into shots, embed sampled keyframes with a CLIP-style model into a shared text-image space, and index them with timestamps. A query is one text embedding and an ANN lookup; rerank only the top candidates with a larger model.
Practise the rest
12 Multimodal questions, 492 in all, each with a model answer. Free.
More from ByteDance interview questions.
More Multimodal interview questions
- What has driven each generation of LLM improvements, how do multimodal LLMs work, and what role does normalisation play in LLMs?
- How does a vision encoder feed images into a multimodal LLM, and where are the bottlenecks?
- Design the training of a multimodal embedding model: architecture, supervision and losses, evaluation metrics and deployment.
- Turn a multimodal model that captions videos into a production service under tight compute and GPU memory limits.
- Design a classifier that assigns genres to games using text, images and tabular metadata, from label quality to serving and monitoring.
- For an offline audio detection system, outline the ML inference and labeling pipeline: feature extraction, keyword spotting, thresholding, calibration and human labeling.