# Hallucination and limits interview questions

> 11 hallucination and limits interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/hallucination
Last updated: 2026-10-23

11 questions: 2 junior, 5 mid-level and 4 senior. Taught in [Hallucination and the Limits of Next-Token Prediction](https://llmdojo.dev/learn/model-foundations/limits).

## [What is a hallucination, and why can a next-token predictor produce one so fluently?](https://llmdojo.dev/interview/q/what-is-a-hallucination-and-why-can-a-next-token-predictor-produce)

Level: Junior

It's **fluent, confident output** that is false or not grounded in any source. The model was trained to predict the token that actually came next in real text, not the true one, and **nothing checks truth**. So a wrong answer reads as smoothly as a right one.

## [GPT-2 small completes 'The capital of Australia is the city of' with Sydney. Walk me through why a model does that.](https://llmdojo.dev/interview/q/gpt-2-small-completes-the-capital-of-australia-is-the-city-of)

Level: Mid-level

Because **likely text isn't true text**. In our run, " Sydney" was the top token at 0.216 and " Canberra" only fifth at 0.0436, and greedy stated Sydney with no hedge.

There's **no lookup**: knowledge is **spread through the weights**, and a plausible wrong answer, the best-known city, can outweigh the right one. We didn't measure why Sydney wins; one plausible reason is that it appears near "Australia" in far more web text.

## [Why do long generated answers leave more room for hallucination than one-word answers?](https://llmdojo.dev/interview/q/why-do-long-generated-answers-leave-more-room-for-hallucination-than-one)

Level: Mid-level

Every generated token **becomes context for the next**, and nothing marks it as the model's own guess, so each step is another chance to go wrong. *AI Engineering* describes **snowballing**: the model keeps justifying an early wrong assumption.

A right start doesn't protect the rest: GPT-2's greedy Everest answer began correctly, then put the summit at about 1,000 feet. That's one reason concise answers seem to help.

## [You want to automatically flag answers that are probably wrong, using the model's token probabilities. How would you design and validate that?](https://llmdojo.dev/interview/q/you-want-to-automatically-flag-answers-that-are-probably-wrong-using-the)

Level: Senior

Treat probability as a **signal to validate**, not a verdict. On our 14 prompts, four right answers, including " Tokyo" at 0.1186, were less probable than the wrong " Sydney" at 0.216, and the GPT-4 report found post-training **weakened calibration**.

So log the probabilities, flag low ones for review, and measure on **labelled traffic** how many real errors the flag catches and how many good answers it wrongly flags. Recheck after any model or prompt change. For free-form answers, Kadavath et al. ask the model whether its answer is true and use that probability.

## [You're shipping a support bot that answers from your company's policy documents. What do you put in place to reduce hallucination, and how do you evaluate it?](https://llmdojo.dev/interview/q/you-re-shipping-a-support-bot-that-answers-from-your-company-s)

Level: Senior

Ground answers in **retrieved policy text**. Context steers strongly: in our toy, one true sentence moved " Canberra" from 0.0436 to 0.8248, though that mostly measures copying, and retrieval can find the wrong text.

So: retrieve, **require a cited passage** for each answer and check it supports the claim, keep answers short and restricted to the given context, and let the bot say it doesn't know. Evaluate **local factual consistency**: is each answer supported by the documents it was given? That's far easier to check than truth against world knowledge.

## [Design how a chatbot falls back when it doesn't know the answer, covering uncertainty, retrieval and tools.](https://llmdojo.dev/interview/q/design-how-a-chatbot-falls-back-when-it-doesn-t-know-the)

Level: Mid-level

Make **abstaining a valid output**. Questions needing fresh or private facts go to retrieval or a tool; if nothing clears a relevance threshold, say so rather than guess. Then check answers: require citations to retrieved passages, and treat disagreement across several samples as an uncertainty signal. Low confidence falls back to a clarifying question or a human. Evaluate on **unanswerable questions** as well as answerable ones.

## [How would you measure an LLM's answer quality and detect hallucinations against the context it was given? Then implement the evaluation component.](https://llmdojo.dev/interview/q/how-would-you-measure-an-llm-s-answer-quality-and-detect-hallucinations)

Level: Mid-level

Quality is task-specific: reference answers where they exist, a rubric judge, human spot checks. Hallucination against supplied context is **faithfulness**: split the answer into claims and check each is **supported by the context**.

```python

from dataclasses import dataclass, field

@dataclass
class Result:
    score: float
    unsupported: list = field(default_factory=list)

def claims(answer):
    parts = re.split(r"(?<=[.!?])\s+", answer)
    return [p.strip() for p in parts if p.strip()]

def faithfulness(answer, context, supports):
    # supports(context, claim) -> bool: an NLI
    # model or a judge prompt
    cs = claims(answer)
    if not cs:
        return Result(1.0)
    bad = [c for c in cs if not supports(context, c)]
    return Result(1 - len(bad) / len(cs), bad)
```

Interviewers look for **claim-level** scoring, a pluggable judge validated against human labels, and the empty case handled.

## [Design a voice assistant whose answers stay grounded. How would you reduce hallucinations?](https://llmdojo.dev/interview/q/design-a-voice-assistant-whose-answers-stay-grounded-how-would-you-reduce)

Level: Senior

Voice adds two problems: **errors start at speech recognition**, and the user can't see sources. So use the recogniser's confidence, and confirm an uncertain name or number before acting on it.

Then answer from tools and retrieval, like calendar, contacts and search, not model memory, telling the model to answer only from those results. Check the draft against them before speaking, and **confirm before any action**. Keep replies short, name the source aloud, and say when it doesn't know. Evaluate on recorded, noisy, accented queries, scoring faithfulness and task success.

## [Users ask about events after the model's training cutoff. How should it recognise those questions and say so?](https://llmdojo.dev/interview/q/users-ask-about-events-after-the-model-s-training-cutoff-how-should)

Level: Mid-level

The model doesn't know today's date, so **tell it**: put the date and the training cutoff in the system prompt.

Then train the behaviour. Fine-tune on questions about things that change, like recent events or latest versions, with answers that give what held at the cutoff and flag it may be stale, or call search. Include stable questions so it doesn't **over-refuse**. Evaluate on dated questions either side of the cutoff.

## [How would you use LLMs and vision-language models in autonomous driving, and could an LLM actually drive the car given hallucination and real-time latency limits?](https://llmdojo.dev/interview/q/how-would-you-use-llms-and-vision-language-models-in-autonomous-driving)

Level: Senior

Not in the control loop. Control runs many times a second and must be verifiable; an LLM's latency varies, and a **hallucinated object or rule** has no safe failure mode.

Use them **off the hot path**: auto-labelling and mining rare scenarios from fleet logs, generating simulation scenarios, explaining decisions, and as a slow planner suggesting high-level intent that a fast, verified planner accepts or rejects, as DriveVLM proposes. Anything a VLM contributes online sits behind a **deterministic safety layer** and is validated in simulation first.

## [For a medical conversational AI, how would you source and split data, choose between prompting, fine-tuning and RAG, and evaluate for hallucination and unsafe advice?](https://llmdojo.dev/interview/q/for-a-medical-conversational-ai-how-would-you-source-and-split-data)

Level: Junior

Use **vetted clinical guidelines** and de-identified, consented conversations, split by patient so nothing leaks into the test set. Start with **RAG over trusted guidelines** for current, cited facts; fine-tune only for tone or format. Clinicians check each claim against sources, and a test set of **unsafe-advice** cases, like emergencies, must trigger referral.

## Related
- [All interview questions](https://llmdojo.dev/interview)