Data on EKS Data Pipeline Layout

Data on EKS Data Pipeline Layout An architecture diagram generated by Archify. Apache Airflow · Workflow orchestration · Amazon EKS cluster · Control Apache Airflow Workflow orchestration Control Apache Kafka · Streaming · Amazon EKS cluster · Hub Apache Kafka Streaming Hub Apache Spark · batch / Structured Streaming · Amazon EKS cluster Apache Spark batch / Structured Streaming Apache Flink · Stream processing · Amazon EKS cluster Apache Flink Stream processing schedule jobs read events real-time events Amazon EKS cluster Legend Backend Message bus

Orchestration

  • • Job dependencies, schedules and retries

Events / state

  • • Kafka retention/replay and Flink state recovery

Processing

  • • Spark supports batch and streaming