Mid-level interview questionAsked at ByteDance · interview write-up
Explain GRPO for reinforcement learning on LLMs, and how it differs from PPO.
Model answer
Both maximise reward with a clipped probability ratio. PPO estimates advantages with a learned value model, often policy-sized. GRPO drops it: it samples a group of answers per prompt and uses each answer's reward, normalised by the group's mean and standard deviation, as the advantage.
GRPO also puts the KL penalty in the loss, not the reward. It saves memory and suits checkable rewards like maths, but needs several samples per prompt.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from ByteDance interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.