EKS Infrastructure for vLLM

EKS Infrastructure for vLLM An architecture diagram generated by Archify. Amazon ECR · Container images · AWS Services Amazon ECR Container images CloudWatch · Metrics · logs · AWS Services CloudWatch Metrics · logs IAM · Permissions · roles · AWS Services IAM Permissions · roles Scheduler · kube-scheduler · EKS Control Plane Scheduler kube-scheduler API Server · kube-apiserver · EKS Control Plane API Server kube-apiserver Controller Manager · kube-controller-manager · EKS Control Plane Controller Manager kube-controller-manager GPU Node Group · p5 · p4d · g6 · g5 · Node Groups · VPC Subnets / Security Groups / EFA · vLLM serving GPU Node Group p5 · p4d · g6 · g5 vLLM serving CPU Node Group · c5 · m5 · Node Groups · VPC Subnets / Security Groups / EFA CPU Node Group c5 · m5 FSx for Lustre · optional shared model storage · Storage FSx for Lustre optional shared model storage Amazon S3 · Model repository · Storage Amazon S3 Model repository Amazon EBS · Block volumes · Storage Amazon EBS Block volumes watch pods watch / reconcile place vLLM pods place auxiliary workloads load model weights mount models model download attach volumes image pull Metrics · logs AWS Services EKS Control Plane Node Groups · VPC Subnets / Security Groups / EFA Storage Legend Backend Database Cloud Security

Model preparation

  • • Workers require the same model revision and path.
  • • Distinguish static FSx PV/PVC from dynamic provisioning.

Infrastructure

  • • Validate image, AMI, drivers, GPUs, IAM and storage paths together.