Kubeflow Spark Operator Lifecycle

Kubeflow Spark Operator Lifecycle A workflow diagram generated by Archify. 01 / User 02 / Kubeflow Controller 03 / Kubernetes Admission 04 / Spark Pods on Kubernetes kubectl apply · SparkApplication CR · User kubectl apply SparkApplication CR Controller submits · spark-submit · Kubeflow Controller · control loop Controller submits spark-submit control loop Observe pod states · driver + executor → CR · Kubeflow Controller Observe pod states driver + executor → CR Kubernetes admission · Kubeflow Pod mutator · Kubernetes Admission Kubernetes admission Kubeflow Pod mutator Driver pod starts · placed by Kubernetes · Spark Pods on Kubernetes Driver pod starts placed by Kubernetes Request executors · driver → Kubernetes API · Spark Pods on Kubernetes Request executors driver → Kubernetes API Execute Spark tasks · tasks assigned by driver · Spark Pods on Kubernetes Execute Spark tasks tasks assigned by driver create CR admission + placement Pod watch reconcile / retry Legend User submission Spark Operator Mutating Webhook CR .status write Spark Pods