Spark on EKS Deep Dive
Review baseline: Apache Spark 4.2.0, Kubernetes 1.34 or later
Last reviewed: September 12, 2026
Apache Spark runs batch, SQL, streaming and other distributed data workloads. Native Kubernetes support was introduced in Spark 2.3; client-mode support followed in 2.4. Spark 4.2.0 documents Kubernetes 1.34+ as its prerequisite. Use kubectl compatible with the actual EKS version rather than an old fixed minimum.
Spark can use the Kubernetes capacity/control plane you already operate without a separate Spark Standalone master or YARN ResourceManager. You still operate or provision node capacity, images, identity, networking, storage and observability. YARN ResourceManager/NodeManager are YARN components, not Spark-specific daemons.
Execution responsibilities
In cluster deploy mode, the submitting client asks the Kubernetes API to create the driver pod. Kubernetes admission, scheduling and node kubelets handle pod placement/startup. The Spark driver requests executor pods and coordinates Spark stages/tasks; it does not replace the Kubernetes scheduler.
Executors register and communicate directly with the driver for Spark work. Kubernetes continues to manage their pod lifecycle. In client mode, the driver runs with the submitting application, either in a pod or on another host; it must be reachable from executors. Both modes support Spark applications.
Chapters
- Spark on Kubernetes fundamentals: cluster/client submission, resource mapping, dynamic allocation and decommissioning conditions.
- Spark Operator: distinguish the Apache and Kubeflow operators, their APIs, job lifecycle, submission and monitoring.
- EMR on EKS: virtual clusters, job submission and execution identities; distinguish managed runtime features from EKS capacity operations.
- Performance and cost: shuffle/storage/CPU/memory bottlenecks, suitable node capabilities, Spot recovery, and executor versus node scaling.
- Best practices and security: Kubernetes and AWS identity, data access, event logs/history, metrics, network policy and recovery.
Spark dynamic resource allocation adjusts executors within an application. It is distinct from Kubernetes Dynamic Resource Allocation for devices and from node autoscaling. Decommissioning can reduce recomputation, but cannot guarantee that every block survives a forced termination.