# Fine-tuning and alignment interview questions

> 20 fine-tuning and alignment interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/fine-tuning
Last updated: 2026-10-23

20 questions: 2 junior, 8 mid-level and 10 senior.

## [Prompting, RAG or fine-tuning: how do you choose?](https://llmdojo.dev/interview/q/prompting-rag-or-fine-tuning-how-do-you-choose)

Level: Junior

Start with **prompting**: cheapest, and often enough. Add **RAG** when the model lacks knowledge, especially private or changing facts. **Fine-tune** when it has the knowledge but not the behaviour, like a format, tone or narrow skill that prompt examples can't fix. They combine, and fine-tuning is the costliest to change.

## [What is LoRA, and why is it so much cheaper than full fine-tuning?](https://llmdojo.dev/interview/q/what-is-lora-and-why-is-it-so-much-cheaper-than-full)

Level: Mid-level

LoRA **freezes the original weights** and learns a small low-rank update for chosen matrices: W + BA, where B and A are thin, rank 8 or 16, say.

**Diagram: A LoRA update**

1. The pretrained weight matrix W stays frozen: no gradients, no optimizer state for it.
2. LoRA adds a thin side path. A squeezes the input down to r numbers and B expands it back; only A and B are trained.
3. After training, B·A can be added into W, so the fine-tuned model runs exactly as fast as the original.

Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models. Counts are for one 4,096 × 4,096 matrix at rank 8.

Only A and B train, so gradients and optimizer state are a **tiny fraction** of the model's, and one base model can serve many adapters. After training you can **merge BA into W**, so inference costs nothing extra.

## [You're handed fine-tuning code whose loss behaves oddly. What do you check?](https://llmdojo.dev/interview/q/you-re-handed-fine-tuning-code-whose-loss-behaves-oddly-what-do)

Level: Senior

Start with what the model actually sees: **decode a batch back to text**. Usual suspects: a chat template that differs from the base model's; labels shifted twice or not at all; a loss **not masked to the answer tokens**, so the model learns to predict prompts; padding counted in the loss; new special tokens missing from the tokenizer; a learning rate too high for the batch. Then **overfit ten examples**: if the loss won't approach zero, the bug is in the pipeline, not the data.

## [Compare RLHF with a reward model to DPO.](https://llmdojo.dev/interview/q/compare-rlhf-with-a-reward-model-to-dpo)

Level: Mid-level

Both learn from **preference pairs**, a chosen and a rejected answer. RLHF trains a **reward model** on them, then optimises the policy against it with PPO, held near a reference model: powerful, but several models and a touchy RL loop.

**Diagram: RLHF with a reward model vs DPO**

1. RLHF: fit a reward model to human preferences, then optimise the policy against it with reinforcement learning.
2. DPO: one supervised-style loss on the same pairs. No reward model, no sampling loop, far fewer moving parts.

Ouyang et al. (2022), Training language models to follow instructions with human feedback; Rafailov et al. (2023), Direct Preference Optimization.

**DPO** drops the reward model and the RL: one loss on the pairs raises the chosen answer's probability relative to the reference. Simpler and steadier.

## [Debug a GRPO training loop. What is the probability ratio, and what goes wrong with it?](https://llmdojo.dev/interview/q/debug-a-grpo-training-loop-what-is-the-probability-ratio-and-what)

Level: Senior

GRPO samples a **group of answers** per prompt, scores them, and uses each answer's reward minus the group mean, over the group's standard deviation, as its advantage: no value model needed. The **ratio** is the new policy's probability of a token over the old policy's that sampled it, clipped near 1 as in PPO. Typical bugs: the ratio built from the **wrong log-probs**, such as the reference model's, or not detached; a missing exp of the log difference; padding left unmasked; advantages normalised over the batch instead of the group.

## [Design a platform for fine-tuning LLMs on AWS, covering distributed training and security between tenants.](https://llmdojo.dev/interview/q/design-a-platform-for-fine-tuning-llms-on-aws-covering-distributed-training)

Level: Senior

A control plane accepts jobs, base model, dataset in the customer's S3, method and hyperparameters, validates them and schedules onto GPU pools. Default to **LoRA**: adapters train on few GPUs and many tenants share one served base model. Full fine-tunes use FSDP or ZeRO across EFA-linked nodes, checkpointing to S3 so spot interruptions resume.

Isolation is the hard part: a **per-tenant KMS key** for data and weights, jobs in tenant-scoped IAM roles with no cross-tenant network path, no shared caches holding tenant data, and every access audited.

## [How do you detect and prevent reward hacking during LLM post-training?](https://llmdojo.dev/interview/q/how-do-you-detect-and-prevent-reward-hacking-during-llm-post-training)

Level: Senior

Reward hacking is the policy finding outputs the **reward model scores highly but people wouldn't**: longer answers, sycophancy, a format the grader likes, or edited tests in code tasks. Detect it when reward keeps rising while held-out human or stronger-model judgments flatten; track length and other proxies, and read the top-reward samples.

**Diagram: RLHF with a reward model vs DPO**

1. RLHF: fit a reward model to human preferences, then optimise the policy against it with reinforcement learning.
2. DPO: one supervised-style loss on the same pairs. No reward model, no sampling loop, far fewer moving parts.

Ouyang et al. (2022), Training language models to follow instructions with human feedback; Rafailov et al. (2023), Direct Preference Optimization.

Prevent it with a **KL penalty** to the reference model, early stopping as overoptimisation begins, reward-model ensembles, retraining the reward model on fresh policy samples, and verifiable rewards with graders the policy can't tamper with.

## [Design an asynchronous RL post-training system for a production chat model.](https://llmdojo.dev/interview/q/design-an-asynchronous-rl-post-training-system-for-a-production-chat-model)

Level: Senior

Split **actors from the learner**. Inference workers on a fast serving engine generate rollouts from a recent policy snapshot; reward workers score them with reward models or verifiers; the learner trains from a buffer and pushes weights periodically. Asynchrony keeps GPUs busy despite rollouts of very different lengths, but samples go **off-policy**: track and cap policy lag, and correct with importance ratios and clipping, as in PPO.

Keep a KL penalty to the reference model, version every weight and rollout, and gate releases on held-out helpfulness, safety and regression evals.

## [Explain GRPO for reinforcement learning on LLMs, and how it differs from PPO.](https://llmdojo.dev/interview/q/explain-grpo-for-reinforcement-learning-on-llms-and-how-it-differs-from)

Level: Mid-level

Both maximise reward with a clipped probability ratio. PPO estimates advantages with a **learned value model**, often policy-sized. **GRPO** drops it: it samples a group of answers per prompt and uses each answer's reward, normalised by the **group's mean and standard deviation**, as the advantage.

GRPO also puts the KL penalty in the loss, not the reward. It saves memory and suits checkable rewards like maths, but needs several samples per prompt.

## [Walk me through the main LLM post-training methods: SFT, RLHF and preference optimisation. What are the trade-offs?](https://llmdojo.dev/interview/q/walk-me-through-the-main-llm-post-training-methods-sft-rlhf-and)

Level: Mid-level

**SFT** imitates demonstrations: simple and stable, but can't exceed its data. RLHF trains a reward model on preferences and optimises against it with PPO: it can beat the demos, but is costly, unstable and prone to **reward hacking**. DPO-style methods learn from preference pairs directly: cheap and steady, but offline.

For checkable tasks like maths and code, RL with **verifiable rewards**, such as GRPO, skips the reward model. SFT usually comes first.

## [An enterprise customer wants a 7B model that's better at coding in their codebase. Design the post-training pipeline.](https://llmdojo.dev/interview/q/an-enterprise-customer-wants-a-7b-model-that-s-better-at-coding)

Level: Senior

Start with the target: an **execution-based eval** built from the customer's real tasks, with unit tests that pass or fail, plus public coding benchmarks to catch regressions.

Data: their repositories, internal libraries and review history, scrubbed of secrets, plus synthetic tasks kept only if they run. Then stages: continued pretraining on their code if it's unusual, SFT on instruction and solution pairs, then **RL from execution feedback** or DPO on passing versus failing samples. Gate every stage on the eval and watch for forgetting. LoRA adapters keep many customers on one base.

## [Walk me through an LLM's lifecycle: pretraining, supervised fine-tuning, RL, and how each stage is evaluated.](https://llmdojo.dev/interview/q/walk-me-through-an-llm-s-lifecycle-pretraining-supervised-fine-tuning-rl)

Level: Mid-level

**Pretraining** learns language and knowledge by next-token prediction on trillions of tokens, most of the compute. **SFT** trains on curated prompt-response pairs so the model follows instructions and a format. Then preference or RL training goes beyond imitation: RLHF with a reward model, DPO on preference pairs, or verifiable rewards for maths and code.

Evaluation runs throughout: loss and benchmarks, then instruction-following, safety and preference scores, so no stage undoes the last.

**Diagram: RLHF with a reward model vs DPO**

1. RLHF: fit a reward model to human preferences, then optimise the policy against it with reinforcement learning.
2. DPO: one supervised-style loss on the same pairs. No reward model, no sampling loop, far fewer moving parts.

Ouyang et al. (2022), Training language models to follow instructions with human feedback; Rafailov et al. (2023), Direct Preference Optimization.

## [A model struggles with multi-step reasoning on a specific domain task. How would you improve it?](https://llmdojo.dev/interview/q/a-model-struggles-with-multi-step-reasoning-on-a-specific-domain-task)

Level: Senior

First measure: a domain eval with checkable answers, and read the failures to tell **missing knowledge from bad reasoning**. Missing facts call for retrieval, not training.

Cheap wins next: chain-of-thought with worked domain examples, and self-consistency, sampling several chains and voting. If that isn't enough, train: SFT on reasoning traces from experts or a stronger model, kept only when the answer is correct, then **RL with verifiable rewards**, as in GRPO. Watch for reward hacking and chains that grow without getting better.

## [Explain DPO and how it differs from PPO-based RLHF, then describe how you would construct its preference training data.](https://llmdojo.dev/interview/q/explain-dpo-and-how-it-differs-from-ppo-based-rlhf-then-describe)

Level: Mid-level

PPO-based RLHF trains a reward model, then samples and maximises reward with a KL penalty. DPO solves that objective in closed form, training on preference pairs with a **classification-style loss**: raise the chosen answer's log-ratio over the rejected one, relative to the reference. No reward model or sampling loop.

**Diagram: RLHF with a reward model vs DPO**

1. RLHF: fit a reward model to human preferences, then optimise the policy against it with reinforcement learning.
2. DPO: one supervised-style loss on the same pairs. No reward model, no sampling loop, far fewer moving parts.

Ouyang et al. (2022), Training language models to follow instructions with human feedback; Rafailov et al. (2023), Direct Preference Optimization.

Data: per prompt, sample several responses **from the SFT model itself**; humans or a validated judge pick chosen and rejected. Drop ties, watch length bias.

## [You're handed a noisy dataset for LLM post-training. How do you decide whether it helps, and what evidence would you show?](https://llmdojo.dev/interview/q/you-re-handed-a-noisy-dataset-for-llm-post-training-how-do)

Level: Mid-level

Treat it as an experiment. Fix the base model, recipe and a **held-out eval suite** for target skills plus general regressions. Train three runs: without the data, with it raw, and with it filtered, say by deduplication, a judge or reward-model score, and removing items that overlap the evals. Show the **deltas with confidence intervals**, over a few seeds if you can, broken down by slice. The evidence is a **clean ablation**, not hand-picked examples.

## [What is the difference between a transformer and an embedding model, and between pretraining and fine-tuning?](https://llmdojo.dev/interview/q/what-is-the-difference-between-a-transformer-and-an-embedding-model-and)

Level: Junior

A transformer is an architecture. An **embedding model** is usually a transformer encoder trained to turn a text into one vector, so similar texts sit close; a generative one predicts the next token. **Pretraining** learns general language from unlabeled text; **fine-tuning** adapts those weights to a task with a smaller, targeted dataset.

## [Explain importance sampling in LLM training: likelihood ratios, support assumptions, off-policy data, variance growth and clipping.](https://llmdojo.dev/interview/q/explain-importance-sampling-in-llm-training-likelihood-ratios-support-assumptions-off-policy)

Level: Senior

We want expectations under the current policy but hold samples from another policy, older or different. **Importance sampling** reweights each by the likelihood ratio, current over behaviour probability; for a sequence, the product of per-token ratios, summed in log space.

It's unbiased only if the behaviour policy **covers the support**: wherever the current policy puts mass, the old one must too. As policies drift apart, ratios multiply and **variance explodes**. So PPO and GRPO clip per-token ratios, trading a little bias for stability, and off-policy data must stay near-policy.

## [Walk me through GRPO in depth: how critic-free RL post-trains a reasoning model, how you parallelise the training, and how you design the reward.](https://llmdojo.dev/interview/q/walk-me-through-grpo-in-depth-how-critic-free-rl-post-trains)

Level: Senior

GRPO drops PPO's value network. It samples a group of answers per prompt and uses each answer's **reward minus the group mean, over the group standard deviation**, as its advantage, inside PPO's clipped objective with a KL penalty to a reference model. No critic saves memory.

Rollout workers run data-parallel inference; the trainer shards with FSDP or tensor and pipeline parallelism, syncing weights each step. **Rewards** should be verifiable, like exact answer match, unit tests or a format check, not a learned reward model that's easier to game.

## [Design data-quality checks for an LLM dataset of spreadsheet tasks, and decide whether a pretrained model needs fine-tuning for them.](https://llmdojo.dev/interview/q/design-data-quality-checks-for-an-llm-dataset-of-spreadsheet-tasks-and)

Level: Mid-level

Make every example **executable**: formulas parse, references point to real cells, and the expected output matches what a spreadsheet engine computes. Then deduplicate, check the **test set doesn't overlap training**, balance task types and sheet sizes, and have experts audit a sample. To decide on fine-tuning, evaluate the pretrained model with good prompts and examples on a held-out set first; fine-tune only if failures are systematic, like wrong function syntax.

## [Complete a PyTorch pipeline that learns to predict highlight snippets in text sections from start and end positions, then train and validate it.](https://llmdojo.dev/interview/q/complete-a-pytorch-pipeline-that-learns-to-predict-highlight-snippets-in-text)

Level: Senior

A SQuAD-style head: **two logits per token**, for start and end.

```python

class SpanModel(nn.Module):
    def __init__(self, encoder, hidden):
        super().__init__()
        self.encoder = encoder
        self.head = nn.Linear(hidden, 2)

    def forward(self, ids, attn):
        out = self.encoder(input_ids=ids,
                           attention_mask=attn)
        s, e = self.head(out.last_hidden_state).unbind(-1)
        neg = torch.finfo(s.dtype).min
        pad = attn == 0
        s = s.masked_fill(pad, neg)
        return s, e.masked_fill(pad, neg)

def train_step(model, opt, b):
    model.train()
    s, e = model(b["ids"], b["attn"])
    loss = (F.cross_entropy(s, b["start"])
            + F.cross_entropy(e, b["end"])) / 2
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()
```

Interviewers check that pads are masked, labels are token indices mapped from character offsets, and validation runs under no_grad and eval mode. Decode the best pair with **end not before start** and a length cap, then score **exact match and token F1** against the gold character spans.

## Related
- [All interview questions](https://llmdojo.dev/interview)