Ray Serve 요청 흐름과 3단계 오토스케일링

Ray Serve 요청 흐름과 3단계 오토스케일링 An architecture diagram generated by Archify. Client · HTTP / gRPC · Architecture component Client HTTP / gRPC Serve proxy · HTTP / gRPC routing · Architecture component Serve proxy HTTP / gRPC routing Deployment: 전처리 · actor replicas · Application (Serve 파이프라인) Deployment: 전처리 actor replicas Deployment: 모델 추론 · actor replicas · Application (Serve 파이프라인) Deployment: 모델 추론 actor replicas 응답 · 클라이언트로 반환 · Architecture component 응답 클라이언트로 반환 Serve controller · replica goals / autoscaling · Architecture component Serve controller replica goals / autoscaling Ray / KubeRay · actor placement / Pod capacity · Architecture component Ray / KubeRay actor placement / Pod capacity K8s + node provisioner · Pod placement / EC2 capacity · Architecture component K8s + node provisioner Pod placement / EC2 capacity HTTP / gRPC DeploymentHandle request metrics / targets actor capacity if needed Pod / node capacity if needed Application (Serve 파이프라인)

요청 처리

  • • Proxy/handle의 caller queue와 replica ongoing 요청을 구분
  • • 모델·전처리 deployment는 actor replica이며 Kubernetes Deployment가 아님

조건부 확장

  • • 기존 여유 용량·group 한도·placement·준비 상태에 따라 확장 여부가 달라짐
  • • 새 actor마다 Pod나 EC2가 하나씩 생기는 구조가 아님