Flink Native roles and shareable task slots

Flink Native roles and shareable task slots An architecture diagram generated by Archify. REST API & Web UI · status / job submit · Kubernetes Cluster › JobManager Pod REST API & Web UI status / job submit Kubernetes ResourceManager · requests / releases pods · Kubernetes Cluster › JobManager Pod Kubernetes ResourceManager requests / releases pods Checkpoint Coordinator · triggers snapshot · Kubernetes Cluster › JobManager Pod Checkpoint Coordinator triggers snapshot JobMaster / Scheduler · deploys subtasks · Kubernetes Cluster › JobManager Pod JobMaster / Scheduler deploys subtasks TaskManager 1 · worker process · Kubernetes Cluster › TaskManager Pod 1 TaskManager 1 worker process Task Slot 1 · Sharing / chaining possible · Kubernetes Cluster › TaskManager Pod 1 Task Slot 1 Sharing / chaining possible Task Slot 2 · Sharing / chaining possible · Kubernetes Cluster › TaskManager Pod 1 Task Slot 2 Sharing / chaining possible TaskManager 2 · worker process · Kubernetes Cluster › TaskManager Pod 2 TaskManager 2 worker process Task Slot 1 · Sharing / chaining possible · Kubernetes Cluster › TaskManager Pod 2 Task Slot 1 Sharing / chaining possible Task Slot 2 · Sharing / chaining possible · Kubernetes Cluster › TaskManager Pod 2 Task Slot 2 Sharing / chaining possible Pod allocation via K8s API Pod allocation via K8s API triggers snapshot triggers snapshot deploys subtasks deploys subtasks Kubernetes Cluster JobManager Pod TaskManager Pod 1 TaskManager Pod 2

JobManager

  • • Coordinates execution, checkpoints and recovery
  • • Application main() runs on the JM

TaskManager / Slots

  • • Several operator tasks can share a slot
  • • A slot does not itself isolate CPU

Native ResourceManager

  • • Requests/releases pods through the Kubernetes API
  • • Subject to idle timeouts, capacity and profiles