Flink Operator and Native-mode scaling

Flink Operator and Native-mode scaling An architecture diagram generated by Archify. FlinkDeployment CR · Application / Session cluster · Architecture component FlinkDeployment CR Application / Session cluster FlinkSessionJob CR · Job submitted to a session cluster · Architecture component FlinkSessionJob CR Job submitted to a session cluster Flink Kubernetes Operator · Reconciles CRs, hosts the built-in autoscaler · Architecture component Flink Kubernetes Operator Reconciles CRs, hosts the built-in autoscaler JobManager Pod · Coordinates the job graph · Architecture component JobManager Pod Coordinates the job graph TaskManager Pods · Run parallel vertex instances · Architecture component TaskManager Pods Run parallel vertex instances Autoscaler · Computes per-vertex parallelism · Architecture component Autoscaler Computes per-vertex parallelism Flink REST metrics · Rates · backlog · pressure · Architecture component Flink REST metrics Rates · backlog · pressure watches watches In-place / redeploy Native allocation / tasks emits metrics emits metrics reads Enabled + constraints

Evaluation inputs

  • • Rates, edge ratios and backlog target
  • • Memory/GC pressure, quotas and scaling limits

Applying scaling

  • • Begin with recommendations in observation mode
  • • Validate state recovery and in-place/redeploy paths