Mid-level interview questionAsked at Scale AI, Microsoft · interview write-up
Walk me through the main LLM post-training methods: SFT, RLHF and preference optimisation. What are the trade-offs?
Model answer
SFT imitates demonstrations: simple and stable, but can't exceed its data. RLHF trains a reward model on preferences and optimises against it with PPO: it can beat the demos, but is costly, unstable and prone to reward hacking. DPO-style methods learn from preference pairs directly: cheap and steady, but offline.
For checkable tasks like maths and code, RL with verifiable rewards, such as GRPO, skips the reward model. SFT usually comes first.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from Scale AI interview questions, Microsoft interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.