Scoring against references interview questions

Exact evaluation gives the same verdict every time: code passes its tests or it doesn't; an answer matches or it doesn't. Subjective evaluation depends on the judge, a person or a model, and can change between graders or runs. Prefer exact where the task allows, so a moving score means the system moved.

For code generation: generate k samples per problem; a problem counts as solved if any of them passes all its tests; pass@k is the share of problems solved. It can only rise with k, so always report k: 80% at k = 10 is much weaker than 80% at k = 1.

They measure how much the output looks like the references, not whether it's right. A correct rewording can score low because no reference matches it, and a wrong answer that changes one number can score near the top. References can also be incomplete or wrong. On HumanEval, OpenAI found BLEU similar for correct and incorrect solutions.

When many phrasings are right. Embeddings compare meaning, so a rewording can score high without sharing words, and “the dog bit the man” isn't scored as identical to its reverse. But it's only as good as the embedding model, which may not represent your domain, and it costs compute on every output.

Take the most exact method the task allows.

If the output can be run and checked, use functional correctness. One short right answer: exact match after normalising. Many valid phrasings and good references: lexical or semantic similarity. Otherwise, a judge with a rubric. Then check the metric itself by reading outputs it scores high and low.

Not on its own. Perplexity measures how predictable a text is to a model. It's a rough capability proxy, and useful for spotting contamination, when it's suspiciously low on a benchmark, or odd text. But post-training such as SFT and RLHF typically raises perplexity while making models better at tasks, so judge chat models on task metrics.

AUC is the chance a random positive outscores a random negative, so compute it from average ranks, which counts ties as half.

def auc(y, s):
    order = np.argsort(s, kind="mergesort")
    ss = s[order]
    ranks = np.empty(len(s))
    i = 0
    while i < len(s):          # average rank per tie group
        j = i
        while j + 1 < len(s) and ss[j + 1] == ss[i]:
            j += 1
        ranks[order[i:j + 1]] = (i + j) / 2 + 1
        i = j + 1
    pos = y == 1
    n_pos, n_neg = pos.sum(), (~pos).sum()
    u = ranks[pos].sum() - n_pos * (n_pos + 1) / 2
    return u / (n_pos * n_neg)

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def fit_logreg(X, y, lam=0.1, lr=0.1, steps=2000):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(steps):
        z = X @ w + b
        p = 0.5 * (1 + np.tanh(z / 2))   # stable sigmoid
        g = p - y
        w -= lr * (X.T @ g / len(y) + lam * w)  # no L2 on b
        b -= lr * g.mean()
    return w, b

Interviewers check ties, overflow on large logits, which subtracting the max prevents, and that the bias isn't regularised.

Keep it small: a JSONL dataset, a client that calls Ollama's generate endpoint with temperature fixed, a scorer, and a report.

import json, requests

URL = "http://localhost:11434/api/generate"

def ask(model, prompt):
    body = {"model": model, "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0, "seed": 0}}
    r = requests.post(URL, json=body, timeout=120)
    r.raise_for_status()
    return r.json()["response"]

def norm(s):
    return " ".join(s.lower().split())

def run(model, path):
    rows = [json.loads(l) for l in open(path)]
    hits = 0
    for row in rows:
        ok = norm(ask(model, row["prompt"])) \
            == norm(row["answer"])
        hits += ok
        print(json.dumps({"id": row["id"], "ok": ok}))
    print(f"accuracy {hits / len(rows):.3f} "
          f"n={len(rows)}")

Normalised exact match suits short answers; swap in a numeric tolerance or a judge for free text. Log each output with model and prompt version so runs stay comparable.