# Multimodal models interview questions

> 12 multimodal models interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/multimodal
Last updated: 2026-10-23

12 questions: 1 junior, 5 mid-level and 6 senior.

## [What has driven each generation of LLM improvements, how do multimodal LLMs work, and what role does normalisation play in LLMs?](https://llmdojo.dev/interview/q/what-has-driven-each-generation-of-llm-improvements-how-do-multimodal-llms)

Level: Mid-level

Generations gained from **more and better data** at compute-optimal scale, instruction tuning and preference training, then architecture tweaks like RoPE, grouped-query attention, MoE and RL for reasoning.

Multimodal LLMs run a **vision encoder** over image patches and project its outputs into the LLM's embedding space as extra tokens.

Normalisation keeps activations in range: **pre-norm** placement stabilises deep stacks, and RMSNorm drops mean-centring to save compute.

## [You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?](https://llmdojo.dev/interview/q/you-need-to-deploy-multimodal-models-under-tight-gpu-and-latency-limits)

Level: Senior

Profile first: preprocessing, vision encoder, prefill and decode each take their share. Then cut **visual tokens**, the big lever, by lowering resolution, pooling or merging tokens; quantise weights; cache image embeddings; and run encoder and LLM as **separate stages** with continuous batching, scaled independently.

**Diagram: Request-level versus continuous batching**

1. Two requests are decoded together: A needs 8 more tokens, B needs 64.
2. In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
3. In continuous batching A is returned as soon as its last token is generated.
4. Its seat is filled by a waiting request at the very next step, so the batch stays full.
5. Orca calls this iteration-level scheduling: the batch is decided again before every step.

After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.

For video, do the heavy work **offline**: split into shots, embed sampled keyframes with a CLIP-style model into a shared text-image space, and index them with timestamps. A query is one text embedding and an ANN lookup; rerank only the top candidates with a larger model.

## [How does a vision encoder feed images into a multimodal LLM, and where are the bottlenecks?](https://llmdojo.dev/interview/q/how-does-a-vision-encoder-feed-images-into-a-multimodal-llm-and)

Level: Senior

A vision encoder, usually a ViT trained contrastively like CLIP, cuts the image into patches and embeds each one. A **projector**, an MLP or a cross-attention resampler, maps those embeddings into the LLM's space, and the LLM reads them as tokens.

The main bottleneck is **visual token count**: a 336-pixel image with 14-pixel patches is 576 tokens, and high-resolution tiling multiplies that, inflating prefill and the KV cache. Low fixed resolution loses small text, and contrastive encoders capture weak spatial detail. Fixes: token pooling, dynamic tiling, and fine-tuning the encoder.

## [Design the training of a multimodal embedding model: architecture, supervision and losses, evaluation metrics and deployment.](https://llmdojo.dev/interview/q/design-the-training-of-a-multimodal-embedding-model-architecture-supervision-and-losses)

Level: Mid-level

Two encoders, image and text, project into one shared space. Train on pairs with a **contrastive InfoNCE loss**, as CLIP does: each image must outscore the batch with its own caption, so big batches and **hard negatives** matter.

Evaluate with retrieval recall at k both ways and zero-shot classification. To deploy, index item embeddings for ANN search and encode queries online. Version the index with the model: **embeddings from two versions aren't comparable**.

## [Turn a multimodal model that captions videos into a production service under tight compute and GPU memory limits.](https://llmdojo.dev/interview/q/turn-a-multimodal-model-that-captions-videos-into-a-production-service-under)

Level: Junior

Cut work before cutting the model. **Sample a few keyframes** instead of every frame, and drop near-duplicates. Use a smaller or quantized model, cap length, and **batch requests** to keep the GPU busy. Captions rarely need to be instant, so use an **async queue** and cache per video. Check quality on held-out videos.

## [Design a classifier that assigns genres to games using text, images and tabular metadata, from label quality to serving and monitoring.](https://llmdojo.dev/interview/q/design-a-classifier-that-assigns-genres-to-games-using-text-images-and)

Level: Mid-level

Treat it as **multi-label**, since games often fit several genres. Clean labels first: creator tags are noisy, so audit a sample and flag likely errors with confident learning.

Encode text with a pretrained text encoder, thumbnails with a CLIP-style image encoder, and keep tabular metadata as features, then **late-fuse** them in a small classifier with per-genre thresholds. Score in batch on updates. Monitor per-genre precision on audited samples, and drift as new genres appear.

## [For an offline audio detection system, outline the ML inference and labeling pipeline: feature extraction, keyword spotting, thresholding, calibration and human labeling.](https://llmdojo.dev/interview/q/for-an-offline-audio-detection-system-outline-the-ml-inference-and-labeling)

Level: Senior

Decode and resample the audio, then compute **log-mel spectrograms** over short overlapping windows. A keyword-spotting model, a small CNN or a pretrained audio encoder, scores each window; smooth over time and merge adjacent hits into events.

**Calibrate** scores on held-out data, for example with temperature scaling, then set per-keyword thresholds from precision targets. Send the uncertain band, plus a random sample of confident hits, to **human labellers** with clear guidelines and agreement checks. Their labels retrain the model and reset the thresholds.

## [Design text search over short videos when only about 20% of them have reliable text metadata.](https://llmdojo.dev/interview/q/design-text-search-over-short-videos-when-only-about-20-of-them)

Level: Mid-level

Don't search only the 20%: **make the rest searchable**. Embed frames and queries in a shared space with a CLIP-style model, and generate text from speech recognition, OCR and vision-language captions. Index both: BM25 over real and generated text, and an **ANN index over visual embeddings**, merged and reranked. Fine-tune the dual encoder on search logs of query and watched video. Report **recall at k separately** for videos with and without metadata.

## [Design a production text-to-speech platform for both interactive streaming synthesis and long asynchronous jobs, including quality evaluation and voice authorization.](https://llmdojo.dev/interview/q/design-a-production-text-to-speech-platform-for-both-interactive-streaming-synthesis)

Level: Senior

Two paths over one model fleet. **Streaming**: normalise the text, then generate and vocode audio in small chunks so playback starts fast; these requests get priority on warm GPUs. **Batch**: split long text at sentence boundaries, synthesise in parallel on spare capacity, stitch with consistent prosody, and return through a job API and storage.

Quality: human listening ratings plus automatic checks, such as word error rate from speech recognition and speaker similarity. **Voice authorization**: clone a voice only with the speaker's recorded consent, bind voices to accounts, and watermark output.

## [Design an image generation system end to end: data curation, diffusion versus autoregressive architectures, training objectives, compute planning and evaluation.](https://llmdojo.dev/interview/q/design-an-image-generation-system-end-to-end-data-curation-diffusion-versus)

Level: Senior

**Data**: large sets of image-text pairs filtered for quality, safety, duplicates and licensing, recaptioned by a vision-language model, since detailed captions improve prompt following. **Architecture**: latent diffusion, with a VAE compressing images and a diffusion transformer or U-Net denoising latents conditioned on a text encoder. Autoregressive models over image tokens fit naturally with LLMs but generate token by token. **Objective**: noise prediction or flow matching. Plan compute with small-scale scaling runs before the big one. Evaluate with FID, CLIP score, prompt-alignment benchmarks and human preference studies.

## [Design a multimodal system that detects unsafe user-generated content in real time, with policy-driven actions and human review.](https://llmdojo.dev/interview/q/design-a-multimodal-system-that-detects-unsafe-user-generated-content-in-real)

Level: Senior

Cheapest first. **Hash matching** against known violating media catches repeats instantly. Then fast classifiers per policy category over fused image and text embeddings, including OCR text, score uploads; heavier models run asynchronously on video frames and borderline items.

A **policy layer**, not the model, maps scores to actions per category: block, limit distribution, blur, or queue. Thresholds come from precision and recall at each action. The uncertain band goes to **human review**, prioritized by severity times expected reach; reviewer decisions and appeals become training labels. Track prevalence, precision at each action, and time to action.

## [Explain the forward and reverse processes of a diffusion model, a common training objective, and how generation differs from training.](https://llmdojo.dev/interview/q/explain-the-forward-and-reverse-processes-of-a-diffusion-model-a-common)

Level: Mid-level

The **forward process** is fixed: it adds Gaussian noise over T steps until data becomes noise, with a closed form for any step. The **reverse process** is learned: a network removes noise one step at a time.

```python
# one DDPM training step, noise-prediction loss
t = torch.randint(0, T, (B,))
eps = torch.randn_like(x0)
a = alpha_bar[t].view(B, 1, 1, 1)
xt = a.sqrt() * x0 + (1 - a).sqrt() * eps
loss = F.mse_loss(model(xt, t), eps)
```

Training samples one random step per example. Generation starts from pure noise and runs the **steps sequentially**, so it's slow; DDIM samplers skip steps.

## Related
- [All interview questions](https://llmdojo.dev/interview)