Spark on Kubernetes Fundamentals Quiz
Review deploy modes, responsibility boundaries, resources, dynamic allocation and termination conditions.
1. Which Kubernetes deployment modes and prerequisite apply to Spark 4.2?
Show answer
Both cluster and client modes are supported; Spark 4.2 documents Kubernetes 1.34+. Client mode dates from 2.4 and is not notebook-only. kubectl must be compatible with the actual API server.
2. Who requests executor pods, places/starts pods and assigns Spark tasks?
Show answer
The driver requests executor pods through the API. Kubernetes scheduler chooses nodes and kubelets start containers. The driver's Spark schedulers assign tasks to registered executors. Neither the API server nor driver replaces Kubernetes scheduling.
3. Does removing the YARN layer eliminate infrastructure operations or driver permissions?
Show answer
No. Nodes, Kubernetes, images, networking, storage and observability remain. The submitter's API authentication and driver's namespace RBAC are required. The example uses a separate executor service account; AWS data permissions and Kubernetes RBAC are distinct.
4. Must both dynamic-allocation/shuffle-tracking flags always be explicitly set?
Show answer
No. Enable dynamic allocation, but shuffle tracking already defaults true in 4.2 and dates from 3.0. Stock Kubernetes ESS is unsupported; decommission-based preservation or suitable reliable ShuffleDataIO are alternatives with conditions. Overlapping mechanisms can delay removal.
5. What termination path is added when decommission is enabled?
Show answer
Spark 4.2 injects a preStop hook running /opt/decom.sh. The official image includes it; it signals the executor JVM with SIGPWR and waits. Block migration also needs storage decommission settings plus actual destinations, time and capacity.
6. Does increasing terminationGracePeriodSeconds to 60 guarantee migration of every block?
Show answer
No. Spark 4.2 overrides the template value, so also set spark.kubernetes.executor.terminationGracePeriodSeconds=60s. Both the default 30 seconds and configured 60 include preStop and are only budgets; forced termination, Spot deadlines or insufficient destination capacity can prevent completion.
7. How does allocation.batch.size differ from initial/minimum executor counts?
Show answer
batch.size controls batches of pod requests. Min/initial/max define executor bounds/start count, with existing executor.instances also considered. The example min2/initial3/instances3 starts at three; node provisioning is separate.
8. How should Spark cores, CPU limits, memory and pod templates be interpreted?
Show answer
Cores relate to default CPU requests and executor task capacity; CPU limits require limit.cores. The JVM example's 1 GiB heap plus 384 MiB minimum overhead yields 1408 MiB request/limit. Templates are read by the submitter and composed by Spark, not incomplete pods to kubectl apply.
9. What must still be verified with shuffle tracking and decommissioning?
Show answer
Check tracking/idle timeouts, forced termination, node capacity and recomputation. Client mode also needs reachable driver endpoints and a real driver-pod owner when applicable. Preserving intermediate blocks does not replace checkpoints, driver recovery or end-to-end exactly-once.