Spark cluster mode: Kubernetes placement versus Spark tasks

Spark cluster mode: Kubernetes placement versus Spark tasks A sequence diagram generated by Archify. Create driver Pod Pod spec / binding Place/start pod Request executor Pods Pod specs / bindings Place/start pods Register Assign Spark tasks Results · status Request executor cleanup on normal shutdown 1. Request and place the driver pod 2. Request and place executor pods 3. Spark task execution and cleanup User · spark-submit · Sequence participant User spark-submit Kubernetes API Server · API objects · admission · Sequence participant Kubernetes API Server API objects · admission Pod placement · scheduler + kubelets · Sequence participant Pod placement scheduler + kubelets Driver Pod · Spark task coordination · Sequence participant Driver Pod Spark task coordination Executor Pods · spark.executor.instances · Sequence participant Executor Pods spark.executor.instances Legend request return default message