Batch API
Also called batch APIs
An inference API that returns results later (often hours) in exchange for a lower price, for work that doesn't need an answer straight away.
Interview questions about Batch API
- How do you cut token costs at scale?
Find where the tokens go first: output tokens cost more each, but long prompts can still dominate the bill.
- Design a backend where clients submit LLM prompt jobs and poll or receive callbacks, with queueing, model routing, prompt versioning, idempotency and cost control.
Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry.