Senior interview questionAsked at Anthropic · interview write-up
Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
Model answer
GRPO samples a group of answers per prompt, scores them, and uses each answer's reward minus the group mean, over the group's standard deviation, as its advantage: no value model needed. The ratio is the new policy's probability of a token over the old policy's that sampled it, clipped near 1 as in PPO. Typical bugs: the ratio built from the wrong log-probs, such as the reference model's, or not detached; a missing exp of the log difference; padding left unmasked; advantages normalised over the batch instead of the group.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from Anthropic interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.
- How do you detect and prevent reward hacking during LLM post-training?