Multimodal models interview questions

Generations gained from more and better data at compute-optimal scale, instruction tuning and preference training, then architecture tweaks like RoPE, grouped-query attention, MoE and RL for reasoning.

Multimodal LLMs run a vision encoder over image patches and project its outputs into the LLM's embedding space as extra tokens.

Normalisation keeps activations in range: pre-norm placement stabilises deep stacks, and RMSNorm drops mean-centring to save compute.

Profile first: preprocessing, vision encoder, prefill and decode each take their share. Then cut visual tokens, the big lever, by lowering resolution, pooling or merging tokens; quantise weights; cache image embeddings; and run encoder and LLM as separate stages with continuous batching, scaled independently.

For video, do the heavy work offline: split into shots, embed sampled keyframes with a CLIP-style model into a shared text-image space, and index them with timestamps. A query is one text embedding and an ANN lookup; rerank only the top candidates with a larger model.

A vision encoder, usually a ViT trained contrastively like CLIP, cuts the image into patches and embeds each one. A projector, an MLP or a cross-attention resampler, maps those embeddings into the LLM's space, and the LLM reads them as tokens.

The main bottleneck is visual token count: a 336-pixel image with 14-pixel patches is 576 tokens, and high-resolution tiling multiplies that, inflating prefill and the KV cache. Low fixed resolution loses small text, and contrastive encoders capture weak spatial detail. Fixes: token pooling, dynamic tiling, and fine-tuning the encoder.

Two encoders, image and text, project into one shared space. Train on pairs with a contrastive InfoNCE loss, as CLIP does: each image must outscore the batch with its own caption, so big batches and hard negatives matter.

Evaluate with retrieval recall at k both ways and zero-shot classification. To deploy, index item embeddings for ANN search and encode queries online. Version the index with the model: embeddings from two versions aren't comparable.

Cut work before cutting the model. Sample a few keyframes instead of every frame, and drop near-duplicates. Use a smaller or quantized model, cap length, and batch requests to keep the GPU busy. Captions rarely need to be instant, so use an async queue and cache per video. Check quality on held-out videos.

Treat it as multi-label, since games often fit several genres. Clean labels first: creator tags are noisy, so audit a sample and flag likely errors with confident learning.

Encode text with a pretrained text encoder, thumbnails with a CLIP-style image encoder, and keep tabular metadata as features, then late-fuse them in a small classifier with per-genre thresholds. Score in batch on updates. Monitor per-genre precision on audited samples, and drift as new genres appear.

Decode and resample the audio, then compute log-mel spectrograms over short overlapping windows. A keyword-spotting model, a small CNN or a pretrained audio encoder, scores each window; smooth over time and merge adjacent hits into events.

Calibrate scores on held-out data, for example with temperature scaling, then set per-keyword thresholds from precision targets. Send the uncertain band, plus a random sample of confident hits, to human labellers with clear guidelines and agreement checks. Their labels retrain the model and reset the thresholds.

Don't search only the 20%: make the rest searchable. Embed frames and queries in a shared space with a CLIP-style model, and generate text from speech recognition, OCR and vision-language captions. Index both: BM25 over real and generated text, and an ANN index over visual embeddings, merged and reranked. Fine-tune the dual encoder on search logs of query and watched video. Report recall at k separately for videos with and without metadata.

Two paths over one model fleet. Streaming: normalise the text, then generate and vocode audio in small chunks so playback starts fast; these requests get priority on warm GPUs. Batch: split long text at sentence boundaries, synthesise in parallel on spare capacity, stitch with consistent prosody, and return through a job API and storage.

Quality: human listening ratings plus automatic checks, such as word error rate from speech recognition and speaker similarity. Voice authorization: clone a voice only with the speaker's recorded consent, bind voices to accounts, and watermark output.

Data: large sets of image-text pairs filtered for quality, safety, duplicates and licensing, recaptioned by a vision-language model, since detailed captions improve prompt following. Architecture: latent diffusion, with a VAE compressing images and a diffusion transformer or U-Net denoising latents conditioned on a text encoder. Autoregressive models over image tokens fit naturally with LLMs but generate token by token. Objective: noise prediction or flow matching. Plan compute with small-scale scaling runs before the big one. Evaluate with FID, CLIP score, prompt-alignment benchmarks and human preference studies.

Cheapest first. Hash matching against known violating media catches repeats instantly. Then fast classifiers per policy category over fused image and text embeddings, including OCR text, score uploads; heavier models run asynchronously on video frames and borderline items.

A policy layer, not the model, maps scores to actions per category: block, limit distribution, blur, or queue. Thresholds come from precision and recall at each action. The uncertain band goes to human review, prioritized by severity times expected reach; reviewer decisions and appeals become training labels. Track prevalence, precision at each action, and time to action.

The forward process is fixed: it adds Gaussian noise over T steps until data becomes noise, with a closed form for any step. The reverse process is learned: a network removes noise one step at a time.

# one DDPM training step, noise-prediction loss
t = torch.randint(0, T, (B,))
eps = torch.randn_like(x0)
a = alpha_bar[t].view(B, 1, 1, 1)
xt = a.sqrt() * x0 + (1 - a).sqrt() * eps
loss = F.mse_loss(model(xt, t), eps)

Training samples one random step per example. Generation starts from pure noise and runs the steps sequentially, so it's slow; DDIM samplers skip steps.