Five roles in a data pipeline

Five roles in a data pipeline A data-flow diagram generated by Archify. 01 / Sources 02 / Ingestion 03 / Storage 04 / Processing 05 / Consumption Apps · DBs · OLTP · logs · 01 / Sources Apps · DBs OLTP · logs IoT · External APIs · event producers · 01 / Sources IoT · External APIs event producers Batch Connectors · periodic loads · 02 / Ingestion Batch Connectors periodic loads Streaming Queue · Kafka · Kinesis · 02 / Ingestion Streaming Queue Kafka · Kinesis Data Lake · raw / curated retention · 03 / Storage Data Lake raw / curated retention Warehouse · SQL transform · 03 / Storage Warehouse SQL transform Batch Processing · Spark · 04 / Processing Batch Processing Spark Stream Processing · Flink · KStreams · 04 / Processing Stream Processing Flink · KStreams Dashboards·Reports · BI · SQL · 05 / Consumption Dashboards·Reports BI · SQL ML · APIs · feature store · 05 / Consumption ML · APIs feature store snapshot / incremental events raw landing retain via connector load, then transform batch transform realtime transform SQL analytics realtime features aggregate output Legend primary data async batch data store data flow