Build a lightweight evaluation harness for a model served through Ollama: run a dataset of prompts, score the answers against references and report the results.

Keep it small: a JSONL dataset, a client that calls Ollama's generate endpoint with temperature fixed, a scorer, and a report.

import json, requests

URL = "http://localhost:11434/api/generate"

def ask(model, prompt):
    body = {"model": model, "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0, "seed": 0}}
    r = requests.post(URL, json=body, timeout=120)
    r.raise_for_status()
    return r.json()["response"]

def norm(s):
    return " ".join(s.lower().split())

def run(model, path):
    rows = [json.loads(l) for l in open(path)]
    hits = 0
    for row in rows:
        ok = norm(ask(model, row["prompt"])) \
            == norm(row["answer"])
        hits += ok
        print(json.dumps({"id": row["id"], "ok": ok}))
    print(f"accuracy {hits / len(rows):.3f} "
          f"n={len(rows)}")

Normalised exact match suits short answers; swap in a numeric tolerance or a judge for free text. Log each output with model and prompt version so runs stay comparable.