# Scoring against references interview questions

> 8 scoring against references interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/reference-scoring
Last updated: 2026-10-23

8 questions: 2 junior, 4 mid-level and 2 senior. Taught in [Scoring Against References: Exact, Lexical and Semantic](https://llmdojo.dev/learn/evaluation-experimentation/reference-scoring).

## [What's the difference between exact and subjective evaluation?](https://llmdojo.dev/interview/q/what-s-the-difference-between-exact-and-subjective-evaluation)

Level: Junior

Exact evaluation gives the **same verdict every time**: code passes its tests or it doesn't; an answer matches or it doesn't. Subjective evaluation depends on the **judge**, a person or a model, and can change between graders or runs. Prefer exact where the task allows, so a moving score means the system moved.

## [What is pass@k?](https://llmdojo.dev/interview/q/what-is-pass-k)

Level: Junior

For code generation: generate **k samples per problem**; a problem counts as solved if **any** of them passes all its tests; pass@k is the share of problems solved. It can only rise with k, so always **report k**: 80% at k = 10 is much weaker than 80% at k = 1.

## [Why can BLEU or ROUGE mislead you when evaluating a model?](https://llmdojo.dev/interview/q/why-can-bleu-or-rouge-mislead-you-when-evaluating-a-model)

Level: Mid-level

They measure how much the output **looks like the references**, not whether it's right. A correct rewording can score low because no reference matches it, and a wrong answer that changes one number can score near the top. References can also be **incomplete or wrong**. On HumanEval, OpenAI found BLEU similar for correct and incorrect solutions.

## [When would you use semantic similarity instead of word overlap, and what does it depend on?](https://llmdojo.dev/interview/q/when-would-you-use-semantic-similarity-instead-of-word-overlap-and-what)

Level: Mid-level

When **many phrasings** are right. Embeddings compare meaning, so a rewording can score high without sharing words, and “the dog bit the man” isn't scored as identical to its reverse. But it's only as good as the **embedding model**, which may not represent your domain, and it costs compute on every output.

## [How do you choose a scoring method for a new LLM feature?](https://llmdojo.dev/interview/q/how-do-you-choose-a-scoring-method-for-a-new-llm-feature)

Level: Senior

Take the **most exact method** the task allows.

**Diagram: Which kind of score fits the task?**

1. If the output can be run and checked, such as code against tests, that's the score to use.
2. If not, but there's one short right answer, compare exactly, after normalising case and spacing.
3. If many phrasings are right and you have good references, compare by shared words or, better, by meaning.
4. If none of those fit, use a judge with a written rubric.
5. And check the metric itself: read a few high-scoring outputs and make sure they're actually good.

If the output can be **run and checked**, use functional correctness. One short right answer: exact match after normalising. Many valid phrasings and good references: lexical or semantic similarity. Otherwise, a **judge with a rubric**. Then check the metric itself by reading outputs it scores high and low.

## [Can perplexity tell you which of two chat models is better?](https://llmdojo.dev/interview/q/can-perplexity-tell-you-which-of-two-chat-models-is-better)

Level: Senior

Not on its own. Perplexity measures how **predictable a text** is to a model. It's a rough capability proxy, and useful for spotting **contamination**, when it's suspiciously low on a benchmark, or odd text. But post-training such as SFT and RLHF typically **raises** perplexity while making models better at tasks, so judge chat models on task metrics.

## [Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.](https://llmdojo.dev/interview/q/implement-auc-roc-with-correct-tie-handling-a-numerically-stable-softmax-and)

Level: Mid-level

AUC is the chance a random positive outscores a random negative, so compute it from **average ranks**, which counts ties as half.

```python
def auc(y, s):
    order = np.argsort(s, kind="mergesort")
    ss = s[order]
    ranks = np.empty(len(s))
    i = 0
    while i < len(s):          # average rank per tie group
        j = i
        while j + 1 < len(s) and ss[j + 1] == ss[i]:
            j += 1
        ranks[order[i:j + 1]] = (i + j) / 2 + 1
        i = j + 1
    pos = y == 1
    n_pos, n_neg = pos.sum(), (~pos).sum()
    u = ranks[pos].sum() - n_pos * (n_pos + 1) / 2
    return u / (n_pos * n_neg)

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def fit_logreg(X, y, lam=0.1, lr=0.1, steps=2000):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(steps):
        z = X @ w + b
        p = 0.5 * (1 + np.tanh(z / 2))   # stable sigmoid
        g = p - y
        w -= lr * (X.T @ g / len(y) + lam * w)  # no L2 on b
        b -= lr * g.mean()
    return w, b
```

Interviewers check **ties**, overflow on large logits, which subtracting the max prevents, and that the **bias isn't regularised**.

## [Build a lightweight evaluation harness for a model served through Ollama: run a dataset of prompts, score the answers against references and report the results.](https://llmdojo.dev/interview/q/build-a-lightweight-evaluation-harness-for-a-model-served-through-ollama-run)

Level: Mid-level

Keep it small: a JSONL dataset, a client that calls Ollama's generate endpoint with **temperature fixed**, a scorer, and a report.

```python

URL = "http://localhost:11434/api/generate"

def ask(model, prompt):
    body = {"model": model, "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0, "seed": 0}}
    r = requests.post(URL, json=body, timeout=120)
    r.raise_for_status()
    return r.json()["response"]

def norm(s):
    return " ".join(s.lower().split())

def run(model, path):
    rows = [json.loads(l) for l in open(path)]
    hits = 0
    for row in rows:
        ok = norm(ask(model, row["prompt"])) \
            == norm(row["answer"])
        hits += ok
        print(json.dumps({"id": row["id"], "ok": ok}))
    print(f"accuracy {hits / len(rows):.3f} "
          f"n={len(rows)}")
```

Normalised exact match suits short answers; swap in a numeric tolerance or a judge for free text. Log each output with model and prompt version so runs stay **comparable**.

## Related
- [All interview questions](https://llmdojo.dev/interview)