Anthropic interview questions

def sdpa(q, k, v, mask=None):  # mask True = may attend
    s = q @ k.transpose(-2, -1) / q.size(-1) ** 0.5
    if mask is not None:
        s = s.masked_fill(~mask, float("-inf"))
    s = s - s.amax(-1, keepdim=True)
    p = s.exp()
    return (p / p.sum(-1, keepdim=True)) @ v

Scale by √d so scores don't saturate softmax. Subtract the row max before exp so nothing overflows; torch's softmax does this internally. A fully masked row gives NaN, since max minus max is inf minus inf, so guard it. In FP16, compute scores and softmax in FP32. FlashAttention does the same blockwise with a running max.

Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long ones. The scheduler admits work while KV-cache memory allows, using paged allocation, and chunks long prefills so they don't stall decodes.

Route by queue depth and free cache, preferring a replica that holds the prompt prefix. Autoscale on queue length and time to first token, not GPU utilisation, keep warm spares because loading weights is slow, and shed batch traffic first under overload.

If a batch costs 100 ms whatever its size, throughput is batch size divided by 0.1 s, so bigger batches are free until memory runs out. Latency is the wait for the next batch plus 100 ms. A batcher fires when the batch fills or a timeout expires. Bound the queue and reject when full, so latency stays predictable. For long generations, batch per decode step instead.

Three jobs. Admission: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch. Fair scheduling inside each replica: weighted fair queuing across tenants, so one heavy tenant can't fill every batch slot. Routing: choose replicas by load and prefix-cache affinity, so repeated system prompts hit warm KV cache, spilling over when a replica runs hot.

Track time to first token and per-token latency per tenant against SLOs, and under overload shed the lowest priority first with retry-after errors.

Write the three costs down, then compare times.

# d model, f hidden (often 4d), T tokens, bf16
flops = 2 * (2 * T * d * f)      # two matmuls
w_bytes = 2 * d * f * 2          # both weights
act_bytes = T * (2 * d + f) * 2  # in, out, hidden
# tensor parallel over n GPUs: one all-reduce of T*d
comm = 2 * (n - 1) / n * T * d * 2   # bytes per GPU
t_compute = flops / n / peak_flops
t_hbm = (w_bytes / n + act_bytes) / hbm_bw
t_net = comm / link_bw

The largest time binds. With few tokens, as in decode, intensity is roughly T FLOPs per byte, below a GPU's FLOPs-to-bandwidth ratio, so HBM binds; big prefill batches turn compute-bound. Sharding divides compute and weights by n but not the all-reduce, so over slow inter-node links the network binds.

Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry. Jobs queue per tenant and priority, with quotas against starvation.

Workers resolve the prompt version pinned in the request, route to a model by task and budget, and store results; clients poll or get a signed webhook. Control cost with token estimates at admission, budget caps, batch APIs for non-urgent work, prompt caching and per-tenant metering. Failures back off, then dead-letter.

Don't parse generated text; read the next-token distribution after the prompt.

import torch

def p_yes(logits, yes_ids, no_ids):
    # logits: (V,) next-token logits
    lp = logits.log_softmax(-1)
    ly = torch.logsumexp(lp[yes_ids], -1)
    ln = torch.logsumexp(lp[no_ids], -1)
    return torch.sigmoid(ly - ln)

Each label has several token spellings, so combine each group with log-sum-exp, then renormalise over the two groups: a sigmoid of the difference. Threshold it after calibrating on labelled data, since raw probabilities are often skewed, and check token ids, because a leading space changes them.

GRPO samples a group of answers per prompt, scores them, and uses each answer's reward minus the group mean, over the group's standard deviation, as its advantage: no value model needed. The ratio is the new policy's probability of a token over the old policy's that sampled it, clipped near 1 as in PPO. Typical bugs: the ratio built from the wrong log-probs, such as the reference model's, or not detached; a missing exp of the log difference; padding left unmasked; advantages normalised over the batch instead of the group.

Count every live tensor.

# Y = X @ W; X (m, k), W (k, n); bf16 = 2 bytes
fwd = 2 * (m * k + k * n + m * n)
# m = k = n = 65536: 8 GiB each, 24 GiB total

Training adds gradients, and mixed-precision Adam needs about 16 bytes per parameter with its FP32 copies. Tensor parallelism splits W itself, so each GPU holds a slice and every step needs an all-reduce or all-gather, best over NVLink. Pipeline parallelism gives whole layers to stages and passes only activations, fine across nodes but with bubbles. It can't split a single matmul too big for one GPU; tensor parallelism can.

size = 500e9                  # bytes
link = 10e9 / 8               # 1.25e9 bytes per second
t_floor = size / link         # 400 s per receiver
t_naive = 1000 * t_floor      # 4e5 s from one source

Each worker must receive 500 GB through its own 10 Gbps link, so 400 seconds is the floor; serving everyone from one source takes about 4.6 days. To approach the floor, every receiver also uploads: split weights into chunks and pipeline them down chains or trees, or swarm BitTorrent-style. Time becomes roughly 400 s plus a few chunk times per hop. Respect topology: one copy crosses each rack uplink, then fans out locally. Checksum chunks and resume on failure.