Part 5: Best Practices and Security Quiz
Spark 4.2.0 / Hadoop 3.5.0 / AWS SDK v2 2.35.4.
1. Which Hadoop/S3A versions match the reviewed Spark 4.2 distribution?
Show answer
Hadoop client and hadoop-aws both use 3.5.0, with AWS SDK v2 bundle 2.35.4 and the resolved runtime dependencies. Recheck bundled versions for another image or EMR.
2. Which IRSA provider fits this SDK v2 combination?
Show answer
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider. The old com.amazonaws.auth.WebIdentityTokenCredentialsProvider is not automatically mapped in this combination.
3. What changes when using Pod Identity?
Show answer
Use ContainerCredentialsProvider, service-account associations, trust for pods.eks.amazonaws.com and Agent/node permissions. IRSA annotations and OIDC setup do not replace this path.
4. Does the default S3A chain automatically use IRSA?
Show answer
The reviewed Hadoop 3.5.0 chain includes the container/instance wrapper but not the web-identity provider. Specify the required provider and verify effective identity and prefix access.
5. Why separate driver, executor and History Server service accounts?
Show answer
To separate driver Kubernetes-management rights from each process's S3 data rights. History Server can start with a read-only role when its cleaner is disabled.
6. Does a namespaced Role restrict a driver to only its own executors?
Show answer
No. It grants resource/verb permissions across the namespace without pod-label ownership conditions. Combine admission and trust boundaries to restrict pod creation, other service accounts and host access.
7. Does ClusterRole always mean cluster-wide permission?
Show answer
No. A RoleBinding referencing a ClusterRole can scope namespaced resource permissions to the binding's namespace. Inspect both rules and the binding type/scope.
8. Does spark.authenticate=true provide Spark UI login?
Show answer
No. It authenticates internal connections. Kubernetes-generated app secrets can be propagated through executor environments, making pod-read permissions important. UI authentication, authorization and TLS are separate.
9. What is wrong with a role-label-only NetworkPolicy?
Show answer
Executors from other jobs in the namespace also match. The example shares a unique run ID between driver/executor labels and policies. Labels are not authentication; consider pod creators able to forge them.
10. Does applying this NetworkPolicy block every other path?
Show answer
Policies are additive and require CNI enforcement. This example limits ingress only. Review egress, node/hostNetwork behavior, other allow policies, DNS/API/AWS endpoints and additional plugins separately.
11. Why combine fixed ports with spark.port.maxRetries=0?
Show answer
A collision fails startup instead of silently moving to a port outside the policy. Configure ports and security separately for additional endpoints.
12. Are PrometheusServlet and the executor aggregate endpoint identical?
Show answer
No. /metrics/prometheus/ serves the driver's Dropwizard registry; /metrics/executors/prometheus/ serves driver-collected executor aggregates. Executors do not each get a Spark UI; compare actual series, labels and units.
13. What is correct about a JMX Java agent and operator chart metrics?
Show answer
The agent runs inside the JVM rather than as another process. Default operator chart metrics describe the operator itself; the chart does not automatically install an agent in every Spark JVM.
14. Is creating a PodMonitor sufficient for collection?
Show answer
Prometheus label/namespace selectors, discovery RBAC, pod ports/paths and NetworkPolicy must agree. Verify UP targets and actual series; short jobs can finish between scrapes.
15. Does retaining the driver pod object retain its UI after the job ends?
Show answer
The live UI stops with its JVM. History Server reconstructs from separately persisted event logs, not stdout/stderr or checkpoints.
16. Does merely mounting a History Server configuration file load it?
Show answer
No. The example explicitly uses spark-class org.apache.spark.deploy.history.HistoryServer --properties-file and runs in the foreground. Verify the path, log directory and S3 identity.
17. Does History Server restore every detail of every past job?
Show answer
Event logs must actually be retained and readable. Missing, corrupt, unflushed, deleted or compacted events limit reconstruction. Event logs are not data-recovery checkpoints or output backups.
18. Are an operator, cluster mode and IRSA the only production choices?
Show answer
No. Choose validated direct submission, an operator or EMR, suitable client/cluster mode and IRSA/Pod Identity for the requirements. Test permissions, data correctness, recovery, networking, resources and costs in the actual environment.