Choosing an Executor: KubernetesExecutor vs. CeleryExecutor

Choosing an Executor: KubernetesExecutor vs. CeleryExecutor An architecture diagram generated by Archify. Scheduler · creates pods via Kubernetes API · KubernetesExecutor Scheduler creates pods via Kubernetes API Pod: task A · Airflow task runtime · KubernetesExecutor Pod: task A Airflow task runtime Pod: task B · Airflow task runtime · KubernetesExecutor Pod: task B Airflow task runtime Pod: task C · Airflow task runtime · KubernetesExecutor Pod: task C Airflow task runtime Scheduler · enqueues tasks onto broker · CeleryExecutor Scheduler enqueues tasks onto broker Celery broker · Redis / RabbitMQ etc. · CeleryExecutor Celery broker Redis / RabbitMQ etc. Pod: celery worker 1 · scalable worker pool · CeleryExecutor Pod: celery worker 1 scalable worker pool Pod: celery worker 2 · scalable worker pool · CeleryExecutor Pod: celery worker 2 scalable worker pool creates 1 pod per task enqueues task worker consumes KubernetesExecutor CeleryExecutor Legend Backend Message bus

KubernetesExecutor

  • • Per-task worker pods; configurable failure retention
  • • Measure startup against image/API/node availability
  • • Control-plane, DB and node costs remain

CeleryExecutor

  • • Warm workers or KEDA scale-to-zero
  • • Configure broker, result backend and concurrency
  • • Deployment or StatefulSet; shared resources/dependencies