Prompt version
Also called prompt versions, prompt versioning
One exact, named revision of a prompt. It is logged with every output it produces so results can be traced and compared; any edit, even a typo fix, makes a new version.
Interview questions about Prompt version
- Design a backend where clients submit LLM prompt jobs and poll or receive callbacks, with queueing, model routing, prompt versioning, idempotency and cost control.
Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry.
- Why version prompts, and what should be logged with every output?
A prompt shapes behaviour like code, so a small edit can change outputs invisibly.
- Build a system to manage prompts and model behaviour in production, then defend its versioning and rollout trade-offs.
Treat prompts as versioned config: a registry stores each prompt's template, model, parameters and output schema as an immutable version that the app…
- Build a lightweight evaluation harness for a model served through Ollama: run a dataset of prompts, score the answers against references and report the results.
Keep it small: a JSONL dataset, a client that calls Ollama's generate endpoint with temperature fixed, a scorer, and a report.
- A RAG assistant gave a wrong answer. How do you find the cause?
Check the steps in order, from the logs. Was the right passage retrieved? If not, fix retrieval. Was it there but ignored or contradicted?
- What is a trace, and what should it capture?
The linked record of one request's path through the system, every step, so a failure can be pinned to the exact step.
- You fall back to a model you've never evaluated with your prompt. What can go wrong?
It's a different model: format reliability, refusals and quality may all change, turning an outage into a quality incident.
- How do you design production metrics for an AI feature?
Start from the failures you want to catch, not what's easy to plot: format failures, factual consistency, safety and refusals, user signals, latency and cost.