Cloud Native Operations

English

한국어

English

한국어

Appearance

Airflow on EKS Deep Dive ​

Review baseline: Airflow 3.3.1 · official Helm chart 1.22.0 Latest upstream patch checked: Airflow 3.3.2 (September 17, 2026); examples remain on 3.3.1 · October 12, 2026

Apache Airflow defines workflow dependencies as DAGs and schedules, runs and observes their tasks. On EKS, per-task pod behavior depends on the executor and operator, not merely on installing a Helm chart.

Airflow 2 reached EOL on April 22, 2026. Use a maintained 3.x release for new deployments. Chart 1.22.0 defaults to Airflow 3.2.2, so verify chart, image and Airflow configuration versions separately. This series reviews 3.3.1 behavior and renders the chart with that version override.

Upstream 3.3.2 was released on September 17, 2026. An explicit Bearer token takes precedence over a session cookie, and an invalid Bearer token rejects the request even when a valid cookie is also present. This can affect API clients that mix browser cookies with service credentials.

This series' examples and Helm checks remain on 3.3.1 and have not been revalidated on 3.3.2.

Core architecture ​

  • The scheduler and executor decide and submit task work and update metadata state.
  • The required DAG processor parses/serializes DAG bundles. Workers also need task code through the relevant bundle or code-distribution path.
  • The API server provides UI, REST API and Task Execution API paths. A supervised Python Task SDK runtime uses that API for state and Connection/Variable/XCom access.
  • The triggerer runs triggers while deferrable tasks wait. It is optional in a minimal deployment that does not use deferral.
  • Metadata backends include PostgreSQL and MySQL. This series uses PostgreSQL as an example, not the only choice. Redis is also not the only Celery broker option.

Airflow control components, metadata, task execution API and alternative executor paths.

Airflow control components, metadata, task execution API and alternative executor paths.Open full screen ↗

DAG bundles do not universally replace git-sync. The official chart still supports git-sync; distinguish delivery from bundle versioning. Parser separation also does not eliminate all latency, database bottlenecks or HA concerns.

Series ​

  1. Airflow architecture on Kubernetes: components, Execution API, databases/brokers and executors.
  2. Helm deployment and executor choice: official chart, versions, connections and worker scaling.
  3. DAG patterns and KubernetesPodOperator: pods, code delivery, bundles and Spark/dbt.
  4. Amazon MWAA integration: managed scope, EKS integration, versions and cost.
  5. Operations and security: HA, migrations, secrets, logs, observation and recovery checks.

Quiz