Component Diagram on Kubernetes

Component Diagram on Kubernetes An architecture diagram generated by Archify. DAG bundle / code · Processor + worker code · Airflow 3: logical components DAG bundle / code Processor + worker code DAG processor · Parse and serialize · Airflow 3: logical components DAG processor Parse and serialize Metadata DB · PostgreSQL / MySQL · Airflow 3: logical components Metadata DB PostgreSQL / MySQL Triggerer (optional) · Async triggers · Airflow 3: logical components Triggerer (optional) Async triggers API server · UI / REST / Execution API · Airflow 3: logical components API server UI / REST / Execution API Scheduler + executor · State and task submission · Airflow 3: logical components Scheduler + executor State and task submission Kubernetes task pod · Task SDK + supervisor · Airflow 3: logical components Kubernetes task pod Task SDK + supervisor Celery worker pool · Task SDK + supervisor · Airflow 3: logical components Celery worker pool Task SDK + supervisor Celery broker · Redis or alternative · Airflow 3: logical components Celery broker Redis or alternative DAG code serialize trigger state state R/W DAG / state R/W queue tasks task delivery Execution API Execution API Kubernetes API: create task pod Airflow 3: logical components Legend Frontend Backend Database Message bus

DAG code

  • • Bundles/code needed by processor and workers
  • • Versioning depends on the bundle

Deployment choices

  • • Database/broker and workload kinds vary
  • • Idle control-plane and node costs remain