Ray Serve Request Flow and Three-Tier Autoscaling

Ray Serve Request Flow and Three-Tier Autoscaling An architecture diagram generated by Archify. Client · HTTP / gRPC · Architecture component Client HTTP / gRPC Serve proxy · HTTP / gRPC routing · Architecture component Serve proxy HTTP / gRPC routing Deployment: Preprocess · actor replicas · Application (Serve pipeline) Deployment: Preprocess actor replicas Deployment: Model Inference · actor replicas · Application (Serve pipeline) Deployment: Model Inference actor replicas Response · returned to client · Architecture component Response returned to client Serve controller · replica goals / autoscaling · Architecture component Serve controller replica goals / autoscaling Ray / KubeRay · actor placement / Pod capacity · Architecture component Ray / KubeRay actor placement / Pod capacity K8s + node provisioner · Pod placement / EC2 capacity · Architecture component K8s + node provisioner Pod placement / EC2 capacity HTTP / gRPC DeploymentHandle request metrics / targets actor capacity if needed Pod / node capacity if needed Application (Serve pipeline)

Request handling

  • • Distinguish caller queues from requests assigned to replicas
  • • Serve deployments use actor replicas, not Kubernetes Deployment objects

Conditional scaling

  • • Existing capacity, group bounds, placement, and readiness affect scaling
  • • A new actor does not imply one new Pod or EC2 node