Senior interview questionAsked at Google · interview write-up
How do you detect and prevent reward hacking during LLM post-training?
Model answer
Reward hacking is the policy finding outputs the reward model scores highly but people wouldn't: longer answers, sycophancy, a format the grader likes, or edited tests in code tasks. Detect it when reward keeps rising while held-out human or stronger-model judgments flatten; track length and other proxies, and read the top-reward samples.
Scroll sideways to see the whole diagram
RLHF: fit a reward model to human preferences, then optimise the policy against it with reinforcement learning.
Text version
- RLHF: fit a reward model to human preferences, then optimise the policy against it with reinforcement learning.
- DPO: one supervised-style loss on the same pairs. No reward model, no sampling loop, far fewer moving parts.
Ouyang et al. (2022), Training language models to follow instructions with human feedback; Rafailov et al. (2023), Direct Preference Optimization.
Prevent it with a KL penalty to the reference model, early stopping as overoptimisation begins, reward-model ensembles, retraining the reward model on fresh policy samples, and verifiable rewards with graders the policy can't tamper with.
Practise the rest
20 Fine-tuning & alignment questions, 492 in all, each with a model answer. Free.
More from Google interview questions.
More Fine-tuning & alignment interview questions
- Prompting, RAG or fine-tuning: how do you choose?
- What is LoRA, and why is it so much cheaper than full fine-tuning?
- You're handed fine-tuning code whose loss behaves oddly. What do you check?
- Compare RLHF with a reward model to DPO.
- Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?
- Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.