Inter-token latency

Also called inter-token latencies

The gap between two consecutive output tokens. Its average is the time per output token.

Interview questions about Inter-token latency

How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
Record one timestamp per token, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total…
Design optimised inference pipelines, and tell me how you would benchmark them.
Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt.