Model Serving Architecture

Model Serving Architecture An architecture diagram generated by Archify. Client · inference request · Architecture component Client inference request Networking Options · Ingress · AWS ALB · API Gateway · Model Serving Architecture Networking Options Ingress · AWS ALB · API Gateway Inference Services · KServe / validated model server · Model Serving Architecture Inference Services KServe / validated model server Model Storage · models: S3/EFS; images: ECR · Model Serving Architecture Model Storage models: S3/EFS; images: ECR Scaling Options · Pods: HPA/KEDA; resources: VPA · Model Serving Architecture Scaling Options Pods: HPA/KEDA; resources: VPA request route Model load Autoscaling Model Serving Architecture Legend Backend Database Cloud External

Access and runtime

  • • Do not publish inference, management and metrics together on an unauthenticated LB.
  • • Validate URI, model format, runtime, credentials and probes together.