Fine-tuning and alignment interview questions

Start with prompting: cheapest, and often enough. Add RAG when the model lacks knowledge, especially private or changing facts. Fine-tune when it has the knowledge but not the behaviour, like a format, tone or narrow skill that prompt examples can't fix. They combine, and fine-tuning is the costliest to change.

LoRA freezes the original weights and learns a small low-rank update for chosen matrices: W + BA, where B and A are thin, rank 8 or 16, say.

Only A and B train, so gradients and optimizer state are a tiny fraction of the model's, and one base model can serve many adapters. After training you can merge BA into W, so inference costs nothing extra.

Start with what the model actually sees: decode a batch back to text. Usual suspects: a chat template that differs from the base model's; labels shifted twice or not at all; a loss not masked to the answer tokens, so the model learns to predict prompts; padding counted in the loss; new special tokens missing from the tokenizer; a learning rate too high for the batch. Then overfit ten examples: if the loss won't approach zero, the bug is in the pipeline, not the data.

Both learn from preference pairs, a chosen and a rejected answer. RLHF trains a reward model on them, then optimises the policy against it with PPO, held near a reference model: powerful, but several models and a touchy RL loop.

DPO drops the reward model and the RL: one loss on the pairs raises the chosen answer's probability relative to the reference. Simpler and steadier.

GRPO samples a group of answers per prompt, scores them, and uses each answer's reward minus the group mean, over the group's standard deviation, as its advantage: no value model needed. The ratio is the new policy's probability of a token over the old policy's that sampled it, clipped near 1 as in PPO. Typical bugs: the ratio built from the wrong log-probs, such as the reference model's, or not detached; a missing exp of the log difference; padding left unmasked; advantages normalised over the batch instead of the group.

A control plane accepts jobs, base model, dataset in the customer's S3, method and hyperparameters, validates them and schedules onto GPU pools. Default to LoRA: adapters train on few GPUs and many tenants share one served base model. Full fine-tunes use FSDP or ZeRO across EFA-linked nodes, checkpointing to S3 so spot interruptions resume.

Isolation is the hard part: a per-tenant KMS key for data and weights, jobs in tenant-scoped IAM roles with no cross-tenant network path, no shared caches holding tenant data, and every access audited.

Reward hacking is the policy finding outputs the reward model scores highly but people wouldn't: longer answers, sycophancy, a format the grader likes, or edited tests in code tasks. Detect it when reward keeps rising while held-out human or stronger-model judgments flatten; track length and other proxies, and read the top-reward samples.

Prevent it with a KL penalty to the reference model, early stopping as overoptimisation begins, reward-model ensembles, retraining the reward model on fresh policy samples, and verifiable rewards with graders the policy can't tamper with.

Split actors from the learner. Inference workers on a fast serving engine generate rollouts from a recent policy snapshot; reward workers score them with reward models or verifiers; the learner trains from a buffer and pushes weights periodically. Asynchrony keeps GPUs busy despite rollouts of very different lengths, but samples go off-policy: track and cap policy lag, and correct with importance ratios and clipping, as in PPO.

Keep a KL penalty to the reference model, version every weight and rollout, and gate releases on held-out helpfulness, safety and regression evals.

Both maximise reward with a clipped probability ratio. PPO estimates advantages with a learned value model, often policy-sized. GRPO drops it: it samples a group of answers per prompt and uses each answer's reward, normalised by the group's mean and standard deviation, as the advantage.

GRPO also puts the KL penalty in the loss, not the reward. It saves memory and suits checkable rewards like maths, but needs several samples per prompt.

SFT imitates demonstrations: simple and stable, but can't exceed its data. RLHF trains a reward model on preferences and optimises against it with PPO: it can beat the demos, but is costly, unstable and prone to reward hacking. DPO-style methods learn from preference pairs directly: cheap and steady, but offline.

For checkable tasks like maths and code, RL with verifiable rewards, such as GRPO, skips the reward model. SFT usually comes first.

Start with the target: an execution-based eval built from the customer's real tasks, with unit tests that pass or fail, plus public coding benchmarks to catch regressions.

Data: their repositories, internal libraries and review history, scrubbed of secrets, plus synthetic tasks kept only if they run. Then stages: continued pretraining on their code if it's unusual, SFT on instruction and solution pairs, then RL from execution feedback or DPO on passing versus failing samples. Gate every stage on the eval and watch for forgetting. LoRA adapters keep many customers on one base.

Pretraining learns language and knowledge by next-token prediction on trillions of tokens, most of the compute. SFT trains on curated prompt-response pairs so the model follows instructions and a format. Then preference or RL training goes beyond imitation: RLHF with a reward model, DPO on preference pairs, or verifiable rewards for maths and code.

Evaluation runs throughout: loss and benchmarks, then instruction-following, safety and preference scores, so no stage undoes the last.

First measure: a domain eval with checkable answers, and read the failures to tell missing knowledge from bad reasoning. Missing facts call for retrieval, not training.

Cheap wins next: chain-of-thought with worked domain examples, and self-consistency, sampling several chains and voting. If that isn't enough, train: SFT on reasoning traces from experts or a stronger model, kept only when the answer is correct, then RL with verifiable rewards, as in GRPO. Watch for reward hacking and chains that grow without getting better.

PPO-based RLHF trains a reward model, then samples and maximises reward with a KL penalty. DPO solves that objective in closed form, training on preference pairs with a classification-style loss: raise the chosen answer's log-ratio over the rejected one, relative to the reference. No reward model or sampling loop.

Data: per prompt, sample several responses from the SFT model itself; humans or a validated judge pick chosen and rejected. Drop ties, watch length bias.

Treat it as an experiment. Fix the base model, recipe and a held-out eval suite for target skills plus general regressions. Train three runs: without the data, with it raw, and with it filtered, say by deduplication, a judge or reward-model score, and removing items that overlap the evals. Show the deltas with confidence intervals, over a few seeds if you can, broken down by slice. The evidence is a clean ablation, not hand-picked examples.

A transformer is an architecture. An embedding model is usually a transformer encoder trained to turn a text into one vector, so similar texts sit close; a generative one predicts the next token. Pretraining learns general language from unlabeled text; fine-tuning adapts those weights to a task with a smaller, targeted dataset.

We want expectations under the current policy but hold samples from another policy, older or different. Importance sampling reweights each by the likelihood ratio, current over behaviour probability; for a sequence, the product of per-token ratios, summed in log space.

It's unbiased only if the behaviour policy covers the support: wherever the current policy puts mass, the old one must too. As policies drift apart, ratios multiply and variance explodes. So PPO and GRPO clip per-token ratios, trading a little bias for stability, and off-policy data must stay near-policy.

GRPO drops PPO's value network. It samples a group of answers per prompt and uses each answer's reward minus the group mean, over the group standard deviation, as its advantage, inside PPO's clipped objective with a KL penalty to a reference model. No critic saves memory.

Rollout workers run data-parallel inference; the trainer shards with FSDP or tensor and pipeline parallelism, syncing weights each step. Rewards should be verifiable, like exact answer match, unit tests or a format check, not a learned reward model that's easier to game.

Make every example executable: formulas parse, references point to real cells, and the expected output matches what a spreadsheet engine computes. Then deduplicate, check the test set doesn't overlap training, balance task types and sheet sizes, and have experts audit a sample. To decide on fine-tuning, evaluate the pretrained model with good prompts and examples on a held-out set first; fine-tune only if failures are systematic, like wrong function syntax.

A SQuAD-style head: two logits per token, for start and end.

import torch
import torch.nn as nn
import torch.nn.functional as F

class SpanModel(nn.Module):
    def __init__(self, encoder, hidden):
        super().__init__()
        self.encoder = encoder
        self.head = nn.Linear(hidden, 2)

    def forward(self, ids, attn):
        out = self.encoder(input_ids=ids,
                           attention_mask=attn)
        s, e = self.head(out.last_hidden_state).unbind(-1)
        neg = torch.finfo(s.dtype).min
        pad = attn == 0
        s = s.masked_fill(pad, neg)
        return s, e.masked_fill(pad, neg)

def train_step(model, opt, b):
    model.train()
    s, e = model(b["ids"], b["attn"])
    loss = (F.cross_entropy(s, b["start"])
            + F.cross_entropy(e, b["end"])) / 2
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()

Interviewers check that pads are masked, labels are token indices mapped from character offsets, and validation runs under no_grad and eval mode. Decode the best pair with end not before start and a length cap, then score exact match and token F1 against the gold character spans.