Parallelism and node capacity: paths and feedback

Parallelism and node capacity: paths and feedback A workflow diagram generated by Archify. 01 / Flink autoscaler 02 / Scale up 03 / Scale down Job decision TM placement Node capacity Vertex metrics · Rates / lag / pressure · Flink autoscaler › Job decision Vertex metrics Rates / lag / pressure Adjust parallelism · Recommendations + constraints · Flink autoscaler › Job decision Adjust parallelism Recommendations + constraints Recovery / lag changes · Next Flink evaluation · Flink autoscaler › Node capacity Recovery / lag changes Next Flink evaluation Request TaskManager pods · Check scheduling feasibility · Scale up › TM placement Request TaskManager pods Check scheduling feasibility Evaluate EC2 capacity · NodePool / quota / availability · Scale up › Node capacity Evaluate EC2 capacity NodePool / quota / availability TM demand decreases · Idle / allocation policy · Scale down › TM placement TM demand decreases Idle / allocation policy Evaluate disruption · Empty / underutilized nodes · Scale down › Node capacity Evaluate disruption Empty / underutilized nodes Scale up Scale down Affects workload Evaluate unschedulable Policy permits

Coupled effects

  • • Flink: parallelism, state recovery and pressure
  • • Node changes can precede job-level decisions

Timings and constraints

  • • Provisioning + restore + backlog catch-up
  • • One timer inequality does not prevent interruptions