Time per output token (TPOT)
Also called time per output token, TPOT
The average time between output tokens after the first one: how fast a streamed reply flows.
Interview questions about Time per output token (TPOT)
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
Record one timestamp per token, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total…
- What does a fair inference benchmark hold fixed?
Everything except the thing you changed: model and precision, prompt and reply lengths, batch, concurrency and arrival rate, warm-up, repeats, environment and…