vLLM Performance Optimization

vLLM Performance Optimization An architecture diagram generated by Archify. EFA Utilization · vpc.amazonaws.com/efa: 1 · Network Optimization EFA Utilization vpc.amazonaws.com/efa: 1 NCCL Settings Optimization · NCCL_DEBUG=INFO · Network Optimization NCCL Settings Optimization NCCL_DEBUG=INFO Node Placement Optimization · topology.kubernetes.io/zone · Network Optimization Node Placement Optimization topology.kubernetes.io/zone GPU Memory Utilization · default0.92 / example0.80 · GPU Memory Optimization GPU Memory Utilization default0.92 / example0.80 Quantization Application · --quantization awq · GPU Memory Optimization Quantization Application --quantization awq CPU / KV offload · separate backends/configuration · GPU Memory Optimization CPU / KV offload separate backends/configuration Batch Size Adjustment · --max-num-batched-tokens=8192 · Throughput Optimization Batch Size Adjustment --max-num-batched-tokens=8192 KV Cache Optimization · --block-size=16 · Throughput Optimization KV Cache Optimization --block-size=16 Tensor Parallel Adjustment · --tensor-parallel-size=8 · Throughput Optimization Tensor Parallel Adjustment --tensor-parallel-size=8 Validate by measurement · throughput / latency / memory · Architecture component Validate by measurement throughput / latency / memory Network Optimization GPU Memory Optimization Throughput Optimization Legend vLLM server option EKS / AWS infrastructure setting Performance improvement (convergence point)

Conditions and limits

  • • kv-cache-memory-bytes overrides automatic KV-budget sizing.
  • • Validate NCCL variables against backend and version.