Two Scaling Loops: Spark DRA and Karpenter

Two Scaling Loops: Spark DRA and Karpenter A workflow diagram generated by Archify. 01 / Spark DRA (task-level decisions) 02 / Kubernetes executor pods 03 / Karpenter (node-level decisions) Scale out Scale in Pending Spark tasks · task backlog grows · Spark DRA (task-level decisions) › Scale out Pending Spark tasks task backlog grows Request more executors · DRA → K8s allocator · Spark DRA (task-level decisions) › Scale out Request more executors DRA → K8s allocator Release idle executors · idle / cache / shuffle · Spark DRA (task-level decisions) › Scale in Release idle executors idle / cache / shuffle New executor pods · Pending · Kubernetes executor pods › Scale out · unschedulable New executor pods Pending unschedulable Executor pods running · scheduled, processing tasks · Kubernetes executor pods › Scale in Executor pods running scheduled, processing tasks Node cleanup candidate · WhenEmpty criteria · Kubernetes executor pods › Scale in Node cleanup candidate WhenEmpty criteria Provision new EC2 node · per NodePool · Karpenter (node-level decisions) › Scale out Provision new EC2 node per NodePool Reclaim the empty node · delay + disruption checks · Karpenter (node-level decisions) › Scale in Reclaim the empty node delay + disruption checks new work arrives policy / delay / budget unschedulable tasks drain, executors idle Legend Kubernetes pod/node state Spark DRA action Karpenter node action