Senior interview questionAsked at Meta · interview write-up
Design an asynchronous RL post-training system for a production chat model.
Model answer
Split actors from the learner. Inference workers on a fast serving engine generate rollouts from a recent policy snapshot; reward workers score them with reward models or verifiers; the learner trains from a buffer and pushes weights periodically. Asynchrony keeps GPUs busy despite rollouts of very different lengths, but samples go off-policy: track and cap policy lag, and correct with importance ratios and clipping, as in PPO.
Keep a KL penalty to the reference model, version every weight and rollout, and gate releases on held-out helpfulness, safety and regression evals.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from Meta interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.