NVIDIA NIM Deployment on EKS

NVIDIA NIM Deployment on EKS An architecture diagram generated by Archify. Approved entry path · authentication / TLS / limits · Ingress Layer Approved entry path authentication / TLS / limits Supported gateway controller · validated route configuration · Ingress Layer Supported gateway controller validated route configuration NIM Container · OpenAI-compatible API :8000 · NIM configuration to validate NIM Container OpenAI-compatible API :8000 Selected NIM backend · 2.0.12: vLLM0.27.1 · NIM configuration to validate Selected NIM backend 2.0.12: vLLM0.27.1 Model Cache · /opt/nim/.cache · NIM configuration to validate Model Cache /opt/nim/.cache p4d.24xlarge · 8x A100 · GPU Node Pool p4d.24xlarge 8x A100 g5.48xlarge · 8x A10G · GPU Node Pool g5.48xlarge 8x A10G Chosen cache storage · local / EBS / shared FS · Model Storage Chosen cache storage local / EBS / shared FS Amazon S3 · Model archive · Model Storage Amazon S3 Model archive NGC Catalog · approved images / models · Model Storage NGC Catalog approved images / models NIM metrics endpoint · :8000/v1/metrics · Monitoring Stack NIM metrics endpoint :8000/v1/metrics Prometheus · Metrics collection · Monitoring Stack Prometheus Metrics collection Grafana Dashboards · Visualization · Monitoring Stack Grafana Dashboards Visualization forward inference load model schedule cache backend configured integration model download expose metrics metric samples query results Ingress Layer NIM configuration to validate GPU Node Pool Model Storage Monitoring Stack

Deployment boundaries

  • • Validate profile, revision, GPUs, cache and probes together.
  • • One shared EBS RWO volume does not fit replicas across nodes.
  • • NGC secret environment delivery differs from a file-only policy.