OpenAI interview questions

Usual planted bugs: the mask applied after softmax, scaling by d instead of head size, a missing transpose before merging heads. For the cache, each decode step computes q, k and v for the new token only, appends k and v, and attends over everything cached.

def step(x, w_qkv, w_o, h, cache):  # x: (B, 1, d)
    B, _, d = x.shape
    q, k, v = (x @ w_qkv).split(d, -1)
    sh = lambda t: t.view(B, 1, h, -1).transpose(1, 2)
    q, k, v = sh(q), sh(k), sh(v)   # (B, h, 1, d/h)
    if "k" in cache:
        k = torch.cat([cache["k"], k], dim=2)
        v = torch.cat([cache["v"], v], dim=2)
    cache["k"], cache["v"] = k, v
    s = q @ k.transpose(-2, -1) / (d // h) ** 0.5
    y = s.softmax(-1) @ v           # no mask: all past
    return y.transpose(1, 2).reshape(B, 1, d) @ w_o

Interviewers check that cached decoding matches the full forward pass and that positions continue from the cache length.

Make abstaining a valid output. Questions needing fresh or private facts go to retrieval or a tool; if nothing clears a relevance threshold, say so rather than guess. Then check answers: require citations to retrieved passages, and treat disagreement across several samples as an uncertainty signal. Low confidence falls back to a clarifying question or a human. Evaluate on unanswerable questions as well as answerable ones.

Treat the context window as a budget. For long documents, chunk along their structure and index chunks plus document summaries, so retrieval goes coarse to fine. For long queries, have a model rewrite them into focused sub-queries, retrieve for each, then merge and rerank.

Pack the best chunks up to a token budget, with the most relevant at the start or end, since models use the middle of long contexts worst. When the answer needs a whole document, use map-reduce: extract per chunk, then answer over the extracts.

Start by splitting the latency budget across retrieval, reranking and generation. I'd use hybrid retrieval, BM25 plus dense vectors, with a tenant filter applied inside the search, never after it. A small cross-encoder reranks, and only the best few chunks enter the prompt, which keeps prefill short.

For cost: cache embeddings and answers per tenant, keep the system prompt a stable prefix for prompt caching, stream tokens, and route easy questions to a smaller model. Track recall and p95 per tenant, with rate limits so no tenant starves the rest.

Permissions first: store each chunk with its source's access-control list and filter by the user's identity inside the retrieval call, so the model never sees what the user can't open. Sync ACL changes fast; stale permissions are a leak.

The agent plans searches, reformulates and opens documents, but every answer must cite passages, and a checker verifies claims against them. With no relevant evidence, or conflicting evidence, it abstains and suggests owners. Evaluate recall, citation accuracy, correct abstention, and permission tests with users of different access.

A control plane accepts jobs, base model, dataset in the customer's S3, method and hyperparameters, validates them and schedules onto GPU pools. Default to LoRA: adapters train on few GPUs and many tenants share one served base model. Full fine-tunes use FSDP or ZeRO across EFA-linked nodes, checkpointing to S3 so spot interruptions resume.

Isolation is the hard part: a per-tenant KMS key for data and weights, jobs in tenant-scoped IAM roles with no cross-tenant network path, no shared caches holding tenant data, and every access audited.

The harness owns everything the model doesn't. It runs the tool-call loop, executes tools in a sandbox with no network by default, scoped credentials and time limits, enforces step and token budgets, and gets human approval for irreversible actions. Every step is logged as a replayable trace.

Evaluation reuses that harness: a versioned task suite with deterministic graders where possible, such as tests passing or final state matching, rubric graders otherwise, and several runs per task because agents are stochastic. Report success, cost, steps and safety violations, and turn production failures into new tasks.

Work in log-probs and subtract the row max before the log-sum-exp, so nothing overflows.

import numpy as np

def masked_ce(logits, y, mask, eps=0.1, T=1.0):
    # logits (N, V), y (N,) ints, mask (N,) 0/1
    z = logits / T
    z = z - z.max(-1, keepdims=True)
    logp = z - np.log(np.exp(z).sum(-1, keepdims=True))
    y = np.where(mask > 0, y, 0)    # ignored ids safe
    nll = -logp[np.arange(len(y)), y]
    loss = (1 - eps) * nll - eps * logp.mean(-1)
    m = mask.astype(loss.dtype)
    return (loss * m).sum() / max(m.sum(), 1.0)

Smoothing mixes the one-hot target with a uniform one, which is a weighted sum of the NLL and the mean negative log-prob. Interviewers check you average over unmasked tokens only, guard an all-masked batch, and never index with ignore labels like -100.

Three stages. Recall: cheap filters for LaTeX or MathML markup, math symbols and known math domains, with extraction that keeps equations, which standard HTML-to-text drops. Precision: a fast classifier such as fastText, trained on seed math pages against random web, rerun with new seeds as DeepSeekMath did. Quality: dedupe, strip benchmark contamination by n-gram matching, and score reasoning content with a small LLM grader.

Validate by training small models on each candidate mix and comparing math benchmarks.