Meta interview questions

Project once, split into heads, attend in each head, merge.

def mhsa(x, w_qkv, w_o, h):            # x: (B, T, d)
    B, T, d = x.shape
    q, k, v = (x @ w_qkv).split(d, -1)
    heads = lambda t: t.view(B, T, h, -1).transpose(1, 2)
    q, k, v = heads(q), heads(k), heads(v)  # (B, h, T, d/h)
    s = q @ k.transpose(-2, -1) / (d // h) ** 0.5
    future = torch.ones(T, T, dtype=torch.bool,
                        device=x.device).triu(1)
    s = s.masked_fill(future, float("-inf"))
    y = s.softmax(-1) @ v
    return y.transpose(1, 2).reshape(B, T, d) @ w_o

Interviewers watch for the mask before softmax, scaling by the head size, not d, and the transpose before the final reshape.

def self_attn(x, wq, wk, wv):        # x: (T, d)
    q, k, v = x @ wq, x @ wk, x @ wv
    s = q @ k.T / q.shape[-1] ** 0.5  # (T, T)
    s = s - s.max(-1, keepdims=True)
    p = np.exp(s)
    return (p / p.sum(-1, keepdims=True)) @ v

Time is O(T² d) and the score matrix takes O(T²) memory, against O(T d²) for projections, so long sequences are attention-dominated. FlashAttention stays exact but tiles the work so the T-by-T matrix never reaches HBM. Sliding-window and sparse patterns make it linear in T; linear and low-rank attention approximate; MQA and GQA shrink the KV cache.

Training: don't train on customer code without opt-in; where you do, dedupe, scrub secrets, and test memorisation by prompting with code prefixes, since models can regurgitate training data verbatim. Inference: prompts land in logs and caches, so scope them per tenant, encrypt them and set retention.

The agent itself is a channel: a poisoned file or issue can tell it to send code out. Sandbox execution with no outbound network by default, allow-list destinations, keep secrets out of context, and require approval for pushes and external calls.

Split actors from the learner. Inference workers on a fast serving engine generate rollouts from a recent policy snapshot; reward workers score them with reward models or verifiers; the learner trains from a buffer and pushes weights periodically. Asynchrony keeps GPUs busy despite rollouts of very different lengths, but samples go off-policy: track and cap policy lag, and correct with importance ratios and clipping, as in PPO.

Keep a KL penalty to the reference model, version every weight and rollout, and gate releases on held-out helpfulness, safety and regression evals.

Let deterministic tools do what they do reliably: linters, type checkers, tests and static analysis run first, and their results go in as facts. The LLM reads the diff with retrieved context, such as callers, related files and the ticket, and comments on intent, logic and design that tools can't see.

Review per file or hunk, then one pass dedupes and ranks findings. Precision beats recall, since noisy bots get ignored: each comment must name a line and a concrete failure. Evaluate on past PRs against comments humans accepted, then track acceptance in production.

Use an agent only where judgement is needed. Classify the ticket into service and issue type, which picks the playbook and tools. Then an investigation loop queries logs, metrics, traces, recent deploys and runbooks through read-only tools, with time windows anchored to the ticket and budgets on steps and tokens.

It returns a diagnosis with cited evidence, a confidence and next steps, which an engineer accepts or corrects. Evaluate on resolved past tickets: root-cause accuracy, whether cited evidence supports the claim, time to diagnosis, and how often engineers accept it.

Combine parallelisms: expert parallelism spreads experts across GPUs with all-to-all token dispatch, tensor parallelism stays inside a node, and pipeline and data parallelism span nodes. The router needs a load-balancing loss and a capacity factor, or a few experts take most tokens; log per-expert load from day one.

All-to-all is the bottleneck, so keep expert groups on the fastest links and overlap dispatch with compute. Add router z-loss for stability, checkpoint often, and validate the data mix on small runs first.

Data: permissively licensed code, deduplicated and filtered, with fill-in-the-middle training so the model completes inside a file, then instruction and repository-level data. Model: a small fast one for inline completion, a larger one for chat and edits. Serving: build context from the cursor's surroundings, open files and retrieved repo snippets, cache the prefix, and cancel stale requests as the user types.

Evaluation: execution-based benchmarks like HumanEval and repo-level tasks offline; acceptance rate and retained code online, through A/B tests.

Data: large sets of image-text pairs filtered for quality, safety, duplicates and licensing, recaptioned by a vision-language model, since detailed captions improve prompt following. Architecture: latent diffusion, with a VAE compressing images and a diffusion transformer or U-Net denoising latents conditioned on a text encoder. Autoregressive models over image tokens fit naturally with LLMs but generate token by token. Objective: noise prediction or flow matching. Plan compute with small-scale scaling runs before the big one. Evaluate with FID, CLIP score, prompt-alignment benchmarks and human preference studies.