Senior interview questionAsked at Google · interview write-up
Explain importance sampling in LLM training: likelihood ratios, support assumptions, off-policy data, variance growth and clipping.
Model answer
We want expectations under the current policy but hold samples from another policy, older or different. Importance sampling reweights each by the likelihood ratio, current over behaviour probability; for a sequence, the product of per-token ratios, summed in log space.
It's unbiased only if the behaviour policy covers the support: wherever the current policy puts mass, the old one must too. As policies drift apart, ratios multiply and variance explodes. So PPO and GRPO clip per-token ratios, trading a little bias for stability, and off-policy data must stay near-policy.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from Google interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.