Part 5: Best Practices and Security Quiz

This quiz tests your understanding of production-readiness best practices for Spark on EKS: IRSA-based S3 access, observability options, the Spark History Server, resource sizing philosophy, and security hardening beyond IAM.

Multiple Choice Questions

1. What class must spark.hadoop.fs.s3a.aws.credentials.provider be set to so driver/executor pods get S3 access via IRSA?
   • A) com.amazonaws.auth.EnvironmentVariableCredentialsProvider
   • B) com.amazonaws.auth.WebIdentityTokenCredentialsProvider
   • C) com.amazonaws.auth.InstanceProfileCredentialsProvider
   • D) org.apache.spark.SparkS3CredentialsProvider

Show Answer

Answer: B) com.amazonaws.auth.WebIdentityTokenCredentialsProvider

Explanation: This provider reads the web identity token that the EKS Pod Identity webhook projects into the pod (based on the service account's eks.amazonaws.com/role-arn annotation) and exchanges it for temporary AWS credentials via STS AssumeRoleWithWebIdentity, refreshing them automatically as they expire. This means no static AWS access keys ever need to exist in a Secret, ConfigMap, or container image.

2. Which Spark configuration keys point the driver and executor pods at an IRSA-bound service account?
   • A) spark.kubernetes.driver.serviceAccountName / spark.kubernetes.executor.serviceAccountName
   • B) spark.kubernetes.authenticate.driver.serviceAccountName / spark.kubernetes.authenticate.executor.serviceAccountName
   • C) spark.hadoop.fs.s3a.serviceAccountName
   • D) spark.driver.iamRole / spark.executor.iamRole

Show Answer

Answer: B) spark.kubernetes.authenticate.driver.serviceAccountName / spark.kubernetes.authenticate.executor.serviceAccountName

Explanation: These spark-submit configuration keys tell Spark which Kubernetes service account to run the driver and executor pods as. When that service account is annotated with eks.amazonaws.com/role-arn, the pods automatically get the projected web identity token needed for IRSA.

3. What is the main operational advantage of Spark's native PrometheusServlet over the JmxSink + JMX Prometheus Exporter pattern?
   • A) It collects a completely different set of metrics
   • B) It requires no external JAR or separate Java agent — it reuses Spark's existing UI port
   • C) It only works with the Spark Operator
   • D) It replaces the need for event logging

Show Answer

Answer: B) It requires no external JAR or separate Java agent — it reuses Spark's existing UI port

Explanation:PrometheusServlet (available since Spark 3.0) exposes Spark's metrics in Prometheus format directly on the existing UI port. The JmxSink + JMX Prometheus Exporter pattern (what the Spark Operator's Helm chart wires up by default) requires an extra -javaagent attached to the JVM and an extra JAR shipped in the image. Both approaches expose the same underlying Spark metrics — they differ in transport and packaging, not in what's measured.

4. When would you deliberately keep using the JmxSink + JMX Prometheus Exporter pattern instead of switching to PrometheusServlet?
   • A) Never — PrometheusServlet is strictly better in every case
   • B) When you want to keep using the Spark Operator's default chart wiring or rely on the broader, more mature ecosystem of community Grafana dashboards built around it
   • C) When running Spark outside of Kubernetes
   • D) When spark.eventLog.enabled is set to false

Show Answer

Answer: B) When you want to keep using the Spark Operator's default chart wiring or rely on the broader, more mature ecosystem of community Grafana dashboards built around it

Explanation: The two approaches trade off differently: PrometheusServlet is operationally simpler (nothing extra to attach to the JVM), while the JMX exporter pattern already has a mature, wide-ranging set of pre-built Grafana dashboards and is what the Spark Operator's chart wires up out of the box. Neither is universally "better" — the choice depends on what you already have running.

5. Why can't you just check the live Spark UI to debug a job that finished last night?
   • A) The Spark UI is disabled by default on Kubernetes
   • B) The driver pod that hosted the UI has likely already been reclaimed by Kubernetes
   • C) The Spark UI only shows currently running stages, never historical ones, regardless of pod lifetime
   • D) Live UIs are only available in client deploy mode

Show Answer

Answer: B) The driver pod that hosted the UI has likely already been reclaimed by Kubernetes

Explanation: Because the driver pod does its own scheduling and hosts the live Spark UI, once a job finishes (or crashes) and Kubernetes reclaims that pod, the UI disappears along with it. This is exactly the gap the Spark History Server fills — it serves a reconstructed UI from persisted event logs regardless of whether the original driver pod still exists.

6. What two Spark configuration properties does the Spark History Server need to read a job's persisted event logs from S3?
   • A) spark.eventLog.enabled and spark.history.fs.logDirectory pointing at the same S3 path used for spark.eventLog.dir
   • B) spark.dynamicAllocation.enabled and spark.decommission.enabled
   • C) spark.kubernetes.authenticate.driver.serviceAccountName only
   • D) spark.metrics.conf and spark.ui.port

Show Answer

Answer: A) spark.eventLog.enabled and spark.history.fs.logDirectory pointing at the same S3 path used for spark.eventLog.dir

Explanation: Jobs must have spark.eventLog.enabled=true and spark.eventLog.dir=s3a://... set so they actually write event logs to S3. The History Server instance then needs spark.history.fs.logDirectory set to that same S3 location (plus its own IRSA-bound service account to read it) so it can periodically rescan and serve a reconstructed UI for completed jobs.

7. According to the resource sizing philosophy recapped in this document, what should driver sizing prioritize?
   • A) Maximum throughput, the same as executors
   • B) Stability — enough headroom that the driver doesn't become a bottleneck or get OOMKilled and take the job down
   • C) Minimum cost, even at the risk of driver instability
   • D) Matching the executor's spark.executor.memory value exactly

Show Answer

Answer: B) Stability — enough headroom that the driver doesn't become a bottleneck or get OOMKilled and take the job down

Explanation: The driver coordinates the job and tracks task state rather than doing the heavy data processing itself, so its sizing goal is stability. Executors, where the actual processing happens, are sized for throughput instead, tuned against the job's real shuffle volume and per-task memory needs. Neither has a universal numeric default — both should come from benchmarking the actual workload.

8. Why does the Spark driver's Kubernetes service account need pod-management permissions at all?
   • A) It doesn't — only executors need pod permissions
   • B) The driver creates, watches, and deletes its own executor pods by calling the Kubernetes API directly
   • C) It's required for S3A credential resolution
   • D) It's only needed when using the Spark Operator, not plain spark-submit

Show Answer

Answer: B) The driver creates, watches, and deletes its own executor pods by calling the Kubernetes API directly

Explanation: As covered in Part 1, the driver acts as its own scheduler on Kubernetes — there's no separate cluster-manager daemon creating executor pods on its behalf. This means the driver's service account needs enough RBAC permissions to manage pods, but the least-privilege version scopes that to a namespaced Role limited to exactly the verbs the driver uses, not a cluster-wide ClusterRole.

9. What is wrong with granting the Spark driver's service account a ClusterRole with broad pod permissions?
   • A) ClusterRole objects don't support RoleBinding
   • B) It violates least privilege — a compromised or misbehaving driver could then affect pods outside its own namespace
   • C) ClusterRole cannot grant create/delete verbs
   • D) It would prevent the driver from creating executor pods at all

Show Answer

Answer: B) It violates least privilege — a compromised or misbehaving driver could then affect pods outside its own namespace

Explanation: The driver only ever needs to manage pods, services, and configmaps within its own namespace. Scoping its permissions to a namespaced Role bound only in that namespace means a compromised or misbehaving driver's blast radius stays contained to that one namespace, instead of being able to reach pods cluster-wide.

10. What is the purpose of pinning spark.driver.port and spark.blockManager.port to fixed values before writing a NetworkPolicy?
    • A) To improve shuffle compression ratios
    • B) So the NetworkPolicy can allow traffic to specific, known ports instead of an unpredictable range
    • C) To enable Dynamic Resource Allocation
    • D) To satisfy IRSA's trust policy requirements

Show Answer

Answer: B) So the NetworkPolicy can allow traffic to specific, known ports instead of an unpredictable range

Explanation: Without pinning these ports, Spark may bind to arbitrary ports, making it impossible to write a NetworkPolicy that allows exactly the ports driver-executor and executor-executor communication needs. Fixing spark.driver.port and spark.blockManager.port (and the executor-side equivalent) to known values lets the policy restrict traffic to just those ports between driver and executor pods.

Short Answer Questions

11. What token does WebIdentityTokenCredentialsProvider rely on, and where does that token come from?

Show Answer

Answer: The web identity token projected into the pod by the EKS Pod Identity webhook, based on the pod's service account annotation (eks.amazonaws.com/role-arn)

Explanation: As long as the service account carries the correct role-arn annotation, EKS automatically projects this token into the pod. WebIdentityTokenCredentialsProvider reads it and calls STS AssumeRoleWithWebIdentity to obtain temporary AWS credentials, refreshing them as they expire — no static credentials required anywhere.

12. Name the two Spark metrics/observability approaches discussed in this document and one reason to pick each.

Show Answer

Answer: (1) Native PrometheusServlet — simpler, no extra agent/JAR, good default for new pipelines. (2) JmxSink + JMX Prometheus Exporter Java agent — worth keeping if already wired up by the Spark Operator's default chart, or to leverage its broader, more mature Grafana dashboard ecosystem.

Explanation: Both expose the same underlying Spark metrics; the choice is about operational overhead versus existing tooling/dashboard investment, not about what gets measured.

13. Why is the Spark History Server necessary given how driver pods behave on Kubernetes?

Show Answer

Answer: Because the driver pod hosts the only live Spark UI, and Kubernetes reclaims that pod once the job finishes — taking the UI with it. The History Server reconstructs a UI from persisted S3 event logs instead of a live driver connection.

Explanation: This is the direct consequence of the driver-does-its-own-scheduling architecture covered in Part 1: no separate long-lived process keeps a job's UI around after the driver pod is gone, so a mechanism that reads durable event logs is the only way to debug a job after the fact.

14. What does spark.dynamicAllocation.shuffleTracking.enabled protect against, and why does this matter more on Kubernetes than on YARN?

Show Answer

Answer: It prevents Dynamic Resource Allocation from removing an executor that's still holding shuffle blocks other tasks need. This matters more on Kubernetes because, unlike YARN's External Shuffle Service, Kubernetes has no daemon that can keep serving shuffle blocks after the executor that produced them is gone.

Explanation: This was introduced in Part 1 as a prerequisite for safely using DRA on Kubernetes, and remains part of the resource-management picture that a production checklist needs to verify before go-live.

15. What two Kubernetes-native mechanisms (beyond IRSA) does this document cover for hardening a Spark-on-EKS deployment, and what does each restrict?

Show Answer

Answer: (1) NetworkPolicy — restricts which pods can reach the driver/executor pods and on which ports. (2) Namespace-scoped RBAC Role/RoleBinding — restricts what the driver's service account can do via the Kubernetes API, limited to its own namespace and only the verbs it needs.

Explanation: IRSA controls access to AWS resources outside the cluster; NetworkPolicy and RBAC control what pods can do to each other and to the Kubernetes API inside the cluster. All three are needed for defense in depth.

Hands-on Questions

16. Write the spark-submit configuration needed so both the driver and executor pods authenticate to S3 via an IRSA-bound service account named spark-s3-sa in namespace spark-jobs.

Show Answer

Answer:

spark-submit \
  --master k8s://https://<EKS_API_SERVER_ENDPOINT>:443 \
  --deploy-mode cluster \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-s3-sa \
  --conf spark.kubernetes.authenticate.executor.serviceAccountName=spark-s3-sa \
  --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider \
  local:///opt/spark/jobs/etl-job.jar

Explanation: Both spark.kubernetes.authenticate.driver.serviceAccountName and the executor equivalent must point at the IRSA-bound service account, and spark.hadoop.fs.s3a.aws.credentials.provider must be set so the S3A connector actually uses the web identity token instead of looking for static credentials.

17. Write a NetworkPolicy that only allows Spark executor pods (spark-role: executor) to reach a driver pod (spark-role: driver) on the driver port (7078) and block manager port (7079), in namespace spark-jobs.

Show Answer

Answer:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: spark-driver-executor-only
  namespace: spark-jobs
spec:
  podSelector:
    matchLabels:
      spark-role: driver
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              spark-role: executor
      ports:
        - protocol: TCP
          port: 7078
        - protocol: TCP
          port: 7079

Explanation: The podSelector targets driver pods, and the single ingress rule only allows traffic from pods labeled spark-role: executor, restricted to the driver port and block manager port. This requires spark.driver.port/spark.driver.blockManager.port to be pinned to fixed values so the policy's port numbers stay valid.

18. Write a namespace-scoped Role and RoleBinding granting the Spark driver's service account (spark-s3-sa in spark-jobs) exactly the permissions it needs to manage its own executor pods, without using a ClusterRole.

Show Answer

Answer:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spark-driver-role
  namespace: spark-jobs
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "get", "list", "watch", "delete"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["create", "get", "list", "watch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spark-driver-rolebinding
  namespace: spark-jobs
subjects:
  - kind: ServiceAccount
    name: spark-s3-sa
    namespace: spark-jobs
roleRef:
  kind: Role
  name: spark-driver-role
  apiGroup: rbac.authorization.k8s.io

Explanation: Using a namespaced Role (not a ClusterRole) bound only within spark-jobs, with verbs limited to what the driver actually does — create/list/watch/delete its own executor pods, read their logs, and manage the headless service/configmaps used for executor discovery — keeps a compromised or misbehaving driver's blast radius contained to its own namespace.

---

Return to Learning Materials | Back to Spark Deep Dive Home