vLLM scaling and capacity boundaries

vLLM scaling and capacity boundaries An architecture diagram generated by Archify. Validated metrics · queue / request / utilization · Architecture component Validated metrics queue / request / utilization One Pod scaling owner · HPA or KEDA / adapter · Architecture component One Pod scaling owner HPA or KEDA / adapter Independent model replicas · readiness / load / drain · Architecture component Independent model replicas readiness / load / drain Node capacity owner · Karpenter or Cluster Autoscaler · Architecture component Node capacity owner Karpenter or Cluster Autoscaler observed signal desired replica count scheduling demand Legend Backend Cloud External

Scaling unit

  • • A TP/PP worker group has a different lifecycle from independent replicas.
  • • Avoid competing scalers or node owners for the same target.

Capacity and cost

  • • Check available GPUs, quotas, AMI/policy and model load time.
  • • EC2 charges can remain after Pod reduction until node termination.