Anatomy of a Modern Data Pipeline — Five Layers

Anatomy of a Modern Data Pipeline — Five Layers A data-flow diagram generated by Archify. 01 / Sources 02 / Ingestion 03 / Storage 04 / Processing 05 / Consumption Apps · DBs · OLTP · logs · 01 / Sources Apps · DBs OLTP · logs IoT · External APIs · event producers · 01 / Sources IoT · External APIs event producers Batch Connectors · periodic loads · 02 / Ingestion Batch Connectors periodic loads Streaming Queue · Kafka · Kinesis · 02 / Ingestion Streaming Queue Kafka · Kinesis Data Lake · raw · low cost · 03 / Storage Data Lake raw · low cost Warehouse · query-optimized · 03 / Storage Warehouse query-optimized Batch Processing · Spark · 04 / Processing Batch Processing Spark Stream Processing · Flink · KStreams · 04 / Processing Stream Processing Flink · KStreams Dashboards·Reports · BI · SQL · 05 / Consumption Dashboards·Reports BI · SQL ML · APIs · feature store · 05 / Consumption ML · APIs feature store snapshots · CDC events raw landing raw retention curated loads (ELT) batch transform realtime transform SQL analytics realtime features Legend primary data async batch data store data flow