How would you measure an LLM's answer quality and detect hallucinations against the context it was given? Then implement the evaluation component.

Quality is task-specific: reference answers where they exist, a rubric judge, human spot checks. Hallucination against supplied context is faithfulness: split the answer into claims and check each is supported by the context.

import re
from dataclasses import dataclass, field

@dataclass
class Result:
    score: float
    unsupported: list = field(default_factory=list)

def claims(answer):
    parts = re.split(r"(?<=[.!?])\s+", answer)
    return [p.strip() for p in parts if p.strip()]

def faithfulness(answer, context, supports):
    # supports(context, claim) -> bool: an NLI
    # model or a judge prompt
    cs = claims(answer)
    if not cs:
        return Result(1.0)
    bad = [c for c in cs if not supports(context, c)]
    return Result(1 - len(bad) / len(cs), bad)

Interviewers look for claim-level scoring, a pluggable judge validated against human labels, and the empty case handled.