Mercor interview questions

Usual planted bugs: the mask applied after softmax, scaling by d instead of head size, a missing transpose before merging heads. For the cache, each decode step computes q, k and v for the new token only, appends k and v, and attends over everything cached.

def step(x, w_qkv, w_o, h, cache):  # x: (B, 1, d)
    B, _, d = x.shape
    q, k, v = (x @ w_qkv).split(d, -1)
    sh = lambda t: t.view(B, 1, h, -1).transpose(1, 2)
    q, k, v = sh(q), sh(k), sh(v)   # (B, h, 1, d/h)
    if "k" in cache:
        k = torch.cat([cache["k"], k], dim=2)
        v = torch.cat([cache["v"], v], dim=2)
    cache["k"], cache["v"] = k, v
    s = q @ k.transpose(-2, -1) / (d // h) ** 0.5
    y = s.softmax(-1) @ v           # no mask: all past
    return y.transpose(1, 2).reshape(B, 1, d) @ w_o

Interviewers check that cached decoding matches the full forward pass and that positions continue from the cache length.

RAG and fine-tuning fill different gaps: RAG supplies changing or citable facts, fine-tuning shapes format, style or a narrow skill. Over the window: retrieve only the relevant chunks, or summarise in a map-reduce pass. Conversation state: keep recent turns verbatim, summarise older ones, and store durable facts outside the prompt. Prompts: version them like code, with templates and a regression suite. Treat retrieved text as data, never instructions, to limit prompt injection.

Keep it small: a JSONL dataset, a client that calls Ollama's generate endpoint with temperature fixed, a scorer, and a report.

import json, requests

URL = "http://localhost:11434/api/generate"

def ask(model, prompt):
    body = {"model": model, "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0, "seed": 0}}
    r = requests.post(URL, json=body, timeout=120)
    r.raise_for_status()
    return r.json()["response"]

def norm(s):
    return " ".join(s.lower().split())

def run(model, path):
    rows = [json.loads(l) for l in open(path)]
    hits = 0
    for row in rows:
        ok = norm(ask(model, row["prompt"])) \
            == norm(row["answer"])
        hits += ok
        print(json.dumps({"id": row["id"], "ok": ok}))
    print(f"accuracy {hits / len(rows):.3f} "
          f"n={len(rows)}")

Normalised exact match suits short answers; swap in a numeric tolerance or a judge for free text. Log each output with model and prompt version so runs stay comparable.

Treat it as an experiment. Fix the base model, recipe and a held-out eval suite for target skills plus general regressions. Train three runs: without the data, with it raw, and with it filtered, say by deduplication, a judge or reward-model score, and removing items that overlap the evals. Show the deltas with confidence intervals, over a few seeds if you can, broken down by slice. The evidence is a clean ablation, not hand-picked examples.

Separate hard filters from soft ranking. Parse profiles and the query into structured fields, like location, experience and skills, with an LLM extractor plus rules, and apply hard criteria as filters so no ranking can break them. Retrieve with hybrid search over profile text, then rank on skill match, seniority and recency. Show why each person matched. Evaluate on recruiter-labelled queries with NDCG, and count hard-criteria violations, which must be zero.