Flink state와 sink commit의 경계

Flink state와 sink commit의 경계 An architecture diagram generated by Archify. MSK / Kafka · 소스 토픽 · Architecture component MSK / Kafka 소스 토픽 Flink runtime · Connector별 version profile · Architecture component Flink runtime Connector별 version profile RocksDB keyed state · Memory / local SST · TaskManagers RocksDB keyed state Memory / local SST S3 checkpoint storage · Shared SST + metadata · Architecture component S3 checkpoint storage Shared SST + metadata Kafka 출력 토픽 · read_committed 컨슈머 · Architecture component Kafka 출력 토픽 read_committed 컨슈머 S3 / Iceberg · Catalog / table commits · Architecture component S3 / Iceberg Catalog / table commits 소비 Keyed state Checkpoint / restore Checkpoint → transaction commit DynamicRecord → table commit TaskManagers

Checkpoint state

  • • Shared SST 참조와 compaction 영향
  • • Storage·retention·restore 조건

Sink commit

  • • Kafka: checkpoint + transaction + read_committed
  • • Sink / table 간 전역 transaction 아님

검증한 version profile

  • • Kafka 5.0.0-2.2 / Flink 2.2.1
  • • Iceberg 1.11.0 / Flink 2.1.3