Senior interview questionAsked at Amazon · interview write-up
Walk me through GRPO in depth: how critic-free RL post-trains a reasoning model, how you parallelise the training, and how you design the reward.
Model answer
GRPO drops PPO's value network. It samples a group of answers per prompt and uses each answer's reward minus the group mean, over the group standard deviation, as its advantage, inside PPO's clipped objective with a KL penalty to a reference model. No critic saves memory.
Rollout workers run data-parallel inference; the trainer shards with FSDP or tensor and pipeline parallelism, syncing weights each step. Rewards should be verifiable, like exact answer match, unit tests or a format check, not a learned reward model that's easier to game.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from Amazon interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.