Spark cluster mode: requests, pod placement and tasks

Spark cluster mode: requests, pod placement and tasks An architecture diagram generated by Archify. spark-submit · local client · Architecture component spark-submit local client Kubernetes API Server · auth · admission · API state · EKS cluster Kubernetes API Server auth · admission · API state Driver Pod · Spark stage/task coordination · EKS cluster Driver Pod Spark stage/task coordination Executor Pod 1 · runs tasks · EKS cluster Executor Pod 1 runs tasks Executor Pod 2 · runs tasks · EKS cluster Executor Pod 2 runs tasks Executor Pod 3 · runs tasks · EKS cluster Executor Pod 3 runs tasks Pod placement and start · scheduler + node kubelets · EKS cluster Pod placement and start scheduler + node kubelets create driver pod place · start request executor pods place · start register · report status Pod specs / bindings EKS cluster