Key LLM Inference Performance Metrics

Key LLM Inference Performance Metrics An architecture diagram generated by Archify. TTFT · first nonempty output received · LLM Inference Metrics TTFT first nonempty output received E2E Latency · Total time from request to completion · LLM Inference Metrics E2E Latency Total time from request to completion ITL · token / chunk intervals · LLM Inference Metrics ITL token / chunk intervals TPOT / TPS · specify window / token count · LLM Inference Metrics TPOT / TPS specify window / token count Throughput · successful completions / time · LLM Inference Metrics Throughput successful completions / time Perceived Responsiveness · First reaction and total wait · User Experience Impact Perceived Responsiveness First reaction and total wait Streaming Quality · Smoothness of token output · User Experience Impact Streaming Quality Smoothness of token output System Capacity · goodput meeting SLOs · User Experience Impact System Capacity goodput meeting SLOs LLM Inference Metrics User Experience Impact Legend User Experience Impact LLM Inference Metrics

Aggregation boundaries

  • • Average TPOT/ITL is undefined for one or fewer tokens.
  • • Aggregate TPS differs from an unweighted request-TPS mean.