Exact match
A scorer that passes an output only if it equals the expected answer exactly. Suits short, exact answers; fails correct answers that are formatted differently.
Interview questions about Exact match
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
JSON valid plus label match on the parsed field. Exact match fails a correct {"label": "billing"} because of the JSON around the word.
- How do you choose a scoring method for a new LLM feature?
Take the most exact method the task allows. If the output can be run and checked, use functional correctness.
- Build a lightweight evaluation harness for a model served through Ollama: run a dataset of prompts, score the answers against references and report the results.
Keep it small: a JSONL dataset, a client that calls Ollama's generate endpoint with temperature fixed, a scorer, and a report.
- A coding agent writes SQL and notebooks over a company's data platform. How would you retrieve and rerank the right tables and notebooks for a request?
Turn each asset into a searchable card: a table's name, columns and comments, tags, owner, sample values and lineage; a notebook's title, cell summary, and…
- Design a question-answering agent over a law firm's legal memos, from ingestion and chunking to grounded, cited answers at scale.
Ingest memos with the metadata lawyers filter on, such as matter, client, jurisdiction and date, plus ethical-wall permissions enforced inside the retrieval…
- Design an MVP RAG assistant that answers employee questions across Jira issues and Confluence pages, covering ingestion, retrieval, ranking and permissions.
Ingest through the product APIs and webhooks so edits and deletions arrive fast.
- Complete a PyTorch pipeline that learns to predict highlight snippets in text sections from start and end positions, then train and validate it.
A SQuAD-style head: two logits per token, for start and end. python import torch import torch.nn as nn import torch.nn.functional as F class…